search

SAEON / somisana

SOMISANA-related tooling
MIT License
6 stars 2 forks source link

Bundle scripts to run the CROCO model into the CROCO Docker image that contains the CROCO executable #3

Closed zachsa closed 2 years ago

zachsa commented 2 years ago

Since the CROCO executable has to be compiled per model, it makes sense to include the bash scripts for invoking the model in the image where the MODEL is compiled.

i.e. this scripts defined HERE should be included in the Docker image build - defined THIS directory

The CROCO image is built per model run now. There are ways of selectively not rebuilding the image if it doesn't exist... but there is no reason to implement this. the CROCO image is built on shared servers, and only takes a couple minutes. The limiting factor in our pipeline is how long the downloads take. My refactoring for parallel curl commands didn't work

image

mattcarr03 commented 2 years ago

Okay, yes that makes sense.

My question then is where are the compiling outputs (croco executable and compile folder with the FORTRAN code) going to be stored as they are needed for each run? Are we going to have a folder which persists outside of the /tmp directory so it is not rewritten?

zachsa commented 2 years ago

It is very quick to compile the croco executable - my current approach is to just build the docker image each run (which means compiling the model each run).

since this can be done in parallel to the download forcing inputs step there is no effect run time.

Old docker images are deleted every week (that are labeled with hash outputs, and not words).

It's possible to specify a step in the workflow file to first check if the image exists and if it does to NOT build the image and recompile CROCO. would you like to implement this @mattcarr03?

zachsa commented 2 years ago

This is complete