Closed AdrienWehrle closed 4 years ago
I'm not going to merge this change into the codebase. I find it overly complicated, and suggest the following. For parallelizing sice.py, I am currently commenting out the call in S3_wrapper.sh and processing mulitple days through the daily mosaic step. Then using GNU parallel I call multiple instances of sice.py like so:
Run on all folders in the mosaic sub-folder:
parallel "python ./sice.py {}" ::: $(find ./mosaic/ -mindepth 1 -type d)
To run on all folders that do not contain O3.tif
(something output by sice.py
), this works:
parallel "python ./sice.py {}" ::: find /sice/Greenland/mosaic/ -mindepth 1 -type d '!' -exec test -e "{}/O3.tif" ';' -print
To run it on the month of June (2018 and 2019):
datelist=$(parallel 'date --iso-8601 -d "{1}-06-01 +{2} days"' ::: 2018 2019 ::: $(seq 0 30))
parallel "python ./sice.py ${mosaic_root}/{}" ::: $datelist