mankoff / SICE

Sentinel 3 Snow Products - GEUS
http://snow.geus.dk/
5 stars 5 forks source link

Optional run of sice.py using multiprocessing #31

Closed AdrienWehrle closed 4 years ago

AdrienWehrle commented 4 years ago
mankoff commented 4 years ago

I'm not going to merge this change into the codebase. I find it overly complicated, and suggest the following. For parallelizing sice.py, I am currently commenting out the call in S3_wrapper.sh and processing mulitple days through the daily mosaic step. Then using GNU parallel I call multiple instances of sice.py like so:

Run on all folders in the mosaic sub-folder:

parallel "python ./sice.py {}" ::: $(find ./mosaic/ -mindepth 1 -type d)

To run on all folders that do not contain O3.tif (something output by sice.py), this works:

parallel "python ./sice.py {}"  ::: find /sice/Greenland/mosaic/ -mindepth 1 -type d '!' -exec test -e "{}/O3.tif" ';' -print

To run it on the month of June (2018 and 2019):

datelist=$(parallel 'date --iso-8601 -d "{1}-06-01 +{2} days"' ::: 2018 2019 ::: $(seq 0 30))
parallel "python ./sice.py ${mosaic_root}/{}" ::: $datelist