search

ai2cm / fv3config

Manipulate FV3GFS run directories
Apache License 2.0
1 stars 0 forks source link

Deprecating run_kubernetes (and maybe run_docker). #92

Open nbren12 opened 4 years ago

nbren12 commented 4 years ago

These functions served us well, but haven't been used in fv3net for a while, mainly because it was difficult to extend run_kubernetes to support I/O patterns involving shared volumes. We have largely replaced use of run_kuberernetes with argo workflows as well as these composable functions for working with containers: https://github.com/VulcanClimateModeling/fv3net/blob/master/external/fv3kube/fv3kube/containers.py. In both cases, we have decomposed the process of running fv3 into a few steps

  1. prepare the full fv3config.yml object from a shorter diff configuration
  2. write the run directory
  3. run the model
  4. post process

Since each of these steps have very different requirements (e.g. 2 and 4 needs auth, but 3 does not) we now run them in separate containers.

IMO, for containerized workflows the role of fv3config is really isolated to step 2, so I think it would make sense to deprecate/remove run_kubernetes. Thoughts?

It is important to support run_native as a legacy option for testing purposes, and I'm not sure about run_docker for local development. I don't really use it, but others might.