issues
search
GeoscienceAustralia
/
dea-cogger
Convert NetCDFs to Cloud Optimized GeoTIFFs
15
stars
4
forks
source link
Prepare for go
#26
Closed
omad
closed
5 years ago
omad
commented
5 years ago
Lots of changes in preparation for large scale runs:
Includes all changes from #25 for improving the work generation step
Detect and fail early if not using full resources in an MPI job
Use MPI in a simpler manner. Split the input file by the number of workers, each MPI worker completes every
n
th
task.
Use
structlog
for more orderly logging
Detect whether running interactively to decide which logging format to use (Needs improvement for within PBS jobs, wrongly detects it as interactive)
Allow skipping pyramids for some bands (
TODO: breaks verify step
)
Clean up code for searching and verifying output COGs
Lots of changes in preparation for large scale runs:
structlog
for more orderly logging