SAR-ARD / s1ard

Sentinel-1 ARD Prototype Processor
MIT License
25 stars 6 forks source link

prevent race conditions in parallel jobs #178

Closed johntruckenbrodt closed 4 months ago

johntruckenbrodt commented 8 months ago

The processor may be run in multiple parallel jobs with overlaps in scene selection. For example:

Similar situations can arise from GRD buffering, which output from multiple scenes and from WBM tiling (see issue #153).

A possible solution could be to introduce file locks (e.g. via the filelock library). However, this might not apply to files written by SNAP and locking a whole folder would be better.

johntruckenbrodt commented 4 months ago

This has mostly been fixed by implementing file locking in https://github.com/SAR-ARD/S1_NRB/pull/192. However, only the most prominent cases have been protected with locked files and other situations might arise in the future for which this has to be implemented as well.