search

VForWaTer / scripts

Notebook collection for different purposes
0 stars 0 forks source link

Lubw restartable #12

Closed AlexDo1 closed 3 years ago

AlexDo1 commented 3 years ago

The LUBW upload script should now be 'restartable'. I don´t know what happens if the upload is interrupted in the middle of the data upload and if some data chunks at the beginning would be uploaded twice in this scenario.

mmaelicke commented 3 years ago

That's ok.

You want me to test it or are you confident it's working fine?

AlexDo1 commented 3 years ago

I'm pretty sure it works fine. The upload is still pretty much the same and I have already tested it locally.

I'm not sure yet if I would interrupt the upload, since I don't know what will happen to the entry whose upload was interrupted. It will already have a datasource, but the data upload is probably not finished yet.

If all data is committed to metacatalog first with session.commit() and session.add(), the upload should be interruptible.