search

ideafast / middleware-services

Python API containing endpoints for smartphone hub applications and transfer to data portal
0 stars 0 forks source link

Skip preparing folders if already prepared #91

Open davidverweij opened 3 years ago

davidverweij commented 3 years ago

Currently, if the prepare stage is completed for a set of Records, but it hasn't been uploaded yet (i.e is_uploaded == False), then the pipeline will still try to execute the prepare stage.

I'd suggest to introduce a quick check if the records are already prepared, and if so, skip the prepare stage; similar to how this is done in the preprocess stage.

This will throw only when the server is being halted in between the prepare and upload stage (which occur sequentially), thus very unlikely. But I ran the pipeline locally with the upload stage uncommented - which did result in this error. More importantly, it kills the overall process completely.

davidverweij commented 3 years ago

Just as an update, this issue just occurred when (most likely) the Data Management Portal performed an update, and the upload stage was halted.