Get AGB workflow running on HPC

Aariq commented 1 month ago

Maybe just needs more cores??

Things to try:

[ ] Give main worker more cores. Maybe it needs additional memory to serialize objects??
[x] Allow workers to access data store directly rather than forcing main worker to handle all the passing of targets objects
[ ] Run just esa_agb target as a test this works locally with 16gb RAM, just takes a long time
[x] Create run.R script that I can submit as a SLURM job so interactive session isn't needed to be up and running for many hours.
[x] Set wall time for heavy-duty workers much higher (like, 24 hours?)

Aariq commented 1 month ago

Bug in geotargets preventing me from using custom launchers for heavy-duty targets: https://github.com/njtierney/geotargets/issues/66

Aariq commented 1 month ago

Might be good idea to split big datasets into tiles then do slopes on each tile with dynamic branching

Aariq commented 1 month ago

Haven't quite gotten it to work, but goign to close this and open a follow-up issue

cct-datascience / organization