Transferring files generated in lake-temperature-model-prep to Caldera

[hcorson-dosch-usgs]

How do we want to get files generated in lake-temperature-model-prep that we need for this pipeline (e.g., 7_config_merge/out/nml_list.rds, final GCM netCDF files) to Caldera?

We could:

• Generate them locally and then manually copy them to Caldera 😬
• Generate them locally, push to S3, and then pull to Caldera
• ? Other options?

[jordansread]

I've used globus to do this when pushing local->caldera, which seems to work pretty well. Is that an option here too?

I think others were also evaluating globus SDK, which is a programmatic way to use globus.

[hcorson-dosch-usgs]

Looks like globus SDK is currently only available for python, but I think using globus to do a manual push would work well for now. I haven't used it before, not sure if @lindsayplatt has -- looks like HPC does have some instructions, though I think those may be out of date, as it sounds like USGS does have a subscription to Globus now. Not sure that it would make sense for this application, but Jeff Falgout recently shared that Globus now has a new feature for scheduled and recurring transfers, which is pretty cool.

[lindsayplatt]

I am not very familiar with globus, but I know that Jake Zwart has used it. We might be able to ping him for some brief guidance.

[jordansread]

new feature for scheduled and recurring transfers

Awesome! That seems like a useful future feature