search

PolarGeospatialCenter / imagery_utils

Other
34 stars 10 forks source link

Update qsub and slurm job files to use python3 env #29

Closed clairecporter closed 1 year ago

clairecporter commented 3 years ago

Move all the qsub and slurm files to loading the pgc conda env. If we assume each user has conda installed and the env is named pgc, that can be hard coded. More elegant might be using the qsub -v option to pass through specific environment variables to the spawned job.

ehusby commented 3 years ago

To rehash my comment from https://github.com/PolarGeospatialCenter/imagery_utils/pull/28:

As we run different scripts on the cluster, let's make an effort to test our pgc conda env on a case-by-case basis. Our pgc env uses GDAL version 3, and while hopefully there aren't changes between GDAL v2 and v3 that break scripts (besides the PROJ axis order thing), we should be careful to watch for any oddities in the results.

To test this in qsub scripts, we'd just replace the module load gdal/2.1.3 line with conda activate pgc.

Maybe we should make a pgc-env branch in the relevant repos that has this change, and we start to use that branch -- make note of which scripts we've tested, and that there are no oddities. Once we've tested (pretty much) everything, then we commit that change to the master branch. Thoughts?