For one of my recent ortho image deliveries, I have source imagery organized into a large number of subfolders. I want to have the output ortho files mirror this organization, but as it stands pgc_ortho.py can only write all output files flat into the destination directory (could be something to rectify). To get around this, I wrapped the call to pgc_ortho.py in a bash loop that changes the source and destination directories to run over each subdirectory, one at a time.
Using the --tasks-per-job argument, I came across an issue where the task bundle text files of an earlier pgc_ortho.py call were being overwritten by the task bundle text files of a later call in the loop! This would happen if one call to pgc_ortho.py in the loop completes in under one second, because all text files generated in one call have a prefix that depends solely on a timestamp to the nearest seconds, such as "Or_src_20201107213423_n.txt" (where n is the number of the job submitted in a single call).
Without going as far as to use Python's standard tempfile module to be ultra sure that each file generated is unique, I thought a simple solution is to add the process ID of the Python call to the bundle file prefix, which should always be unique among multiple Python processing running on the same Nunatak login node. I also condensed the bundle file-writing code into a function and moved it to the utils.py library script.
So far I've tested these changes in ortho tasks not submitted to pbs, submitted to pbs serially, and submitted to pbs in task bundles.
I know the same --tasks-per-job feature is in several other PGC scripts, but it seems it's only in pgc_ortho.py as far as the imagery_utils repo is concerned. Could be something to fix in those other scripts, but this bug should be pretty darn rare.
For one of my recent ortho image deliveries, I have source imagery organized into a large number of subfolders. I want to have the output ortho files mirror this organization, but as it stands
pgc_ortho.pycan only write all output files flat into the destination directory (could be something to rectify). To get around this, I wrapped the call to pgc_ortho.py in a bash loop that changes the source and destination directories to run over each subdirectory, one at a time.Using the
--tasks-per-jobargument, I came across an issue where the task bundle text files of an earlier pgc_ortho.py call were being overwritten by the task bundle text files of a later call in the loop! This would happen if one call to pgc_ortho.py in the loop completes in under one second, because all text files generated in one call have a prefix that depends solely on a timestamp to the nearest seconds, such as "Or_src_20201107213423_n.txt" (wherenis the number of the job submitted in a single call).Without going as far as to use Python's standard
tempfilemodule to be ultra sure that each file generated is unique, I thought a simple solution is to add the process ID of the Python call to the bundle file prefix, which should always be unique among multiple Python processing running on the same Nunatak login node. I also condensed the bundle file-writing code into a function and moved it to theutils.pylibrary script.So far I've tested these changes in ortho tasks not submitted to pbs, submitted to pbs serially, and submitted to pbs in task bundles.
I know the same
--tasks-per-jobfeature is in several other PGC scripts, but it seems it's only inpgc_ortho.pyas far as theimagery_utilsrepo is concerned. Could be something to fix in those other scripts, but this bug should be pretty darn rare.