pgierz
commented
4 years ago ...post-processing is still on my (never shrinking) TODO list. I understand that mixing YAML/Python is likely a hurdle to jump for most people wanting to do post-jobs, especially if we don't have a good example of how to do that. Even more so if the colleagues trying to do this aren't Python fluent. Complete separation of programming logic and model information will be akin to Sisyphus and his boulder...
How about the following:
-
We provide a clean interface for attaching scripts (regardless of language) into the job chain. This is already more-or-less implemented with Dirk's plugin functionality (which is in dire need of documentation!) Currently, it only consumes Python dictionaries.
-
We could (quite easily I guess; it's just a try/except block) provide an interface to alternatively list shell scripts which would be executed. This interface would simply check, in the following order, which criteria apply:
a. Did you list a Python function or method? Yes? Hooray. Do that. We expect config in as the argument, and config out as the return
b. Did you list a (shell) script? This would be checked via filename extension (to keep the programming easier, it might be nice to define files that end in <lalala>.esm_extension...or something similar. We could just try if importing the file in Python works and otherwise assume it's something else) This would be executed via subprocesses.run and receive as arguments anything the YAML recipe would give as arguments/keyword arguments. This would necessitate changing the recipe to a list of dictionaries (for args/kwargs) rather than just a list of strings. Not sure exactly what to do with the config. Just take it in and spit it back out?
c. You did something weird that doesn't fit into the above two options. Abort.
One big downside here is that it opens up a huge gate for people to do whatever they want. Erase the entire experiment tree? sure, why not. Override every file with a cdo random? ok, you can do that... I doubt anyone will actually do that; but still....
For Jan: this would imply the following:
experiment_recipie:
# ....default run stuff....
- /path/to/my/script.sh:
- args:
- toot
- tralala
# ...other steps...and in /path/to/my/script.sh:
# Some default bootstrapping would go here to make sure you get all the variables in YAML
arg1=$1 # Here is toot
arg2=$2 # Here is tralala
cdo info $arg1
joakimkjellsson
dbarbi
Hello together,
I thought I should explain a bit my thoughts on the postprocessing.
Let's step back a bit and look at old-style runscripts like the MPI-ESM from three years ago. Those were quite lengthy files that conflated information about the model, the computer and the experiment with programming of e.g. calender functions, namelist modification, hostfile generation, runtime loop etc. This was fairly inefficient, as users of all levels, including end-users were always confronted with some of the guts at an interface level. This leads to slower progress of setting up experiments and ultimately does a disservice to research in earth system sciences.
The ksh version of the ESM-Tools fixed that by dissecting the runscripts along an information/process boundary. Processes, which were often the same across different models, were moved into the back-end and worked on mainly by core developers (Dirk and Paul), while pure information was placed into the front-end, where this became lists of information, not unlike an extra namelist or the yaml files we use now. Those we called runscripts and they were mainly what scientists/end users saw. In between there was a grey area, a place where the separation of information and programming was not complete, and those were the model and coupling specific .function files. They contained primarily three types of content:
and they were mainly worked on by model developers (Joakim, Martin B., me and others).
With the python esm_tools we are now trying to complete the separation. The default values went into a yaml (I like that) and you try to include even one off processes for model runscripts into the core functionally (more work for you, fine by me). However for post-processing the list of potential candidate methods for inclusion is sheer endless, and so it's not practical to do that in the core. To keep with the theme of separation of information and programming you defined an interface with which model developers create their own post-processing command interface. Model devlopers would then use their own interface and place the post-processing commands in a list to chain them together.
To me it seems that this does not make life easier on the model developer level, as programming the post processing still has to be done anyway, but then additionally it is translated into the yaml interface at a command level and then pieced back together again from the commands defined in the self-made interface. If you were to make this list tool wide, not every command would have to be implemented again for every model, however you would end up with a very long and unwieldy list after a short while, because post-processing task are so diverse. So that too does not sound all the appealing to me.
My hunch is that keeping a small grey zone where info and programming are mixed would actually lead to faster and easier implementation of new post-processing features and models, thus makeing esm_tools a more productive tool for earth systems sciences.
I'm open to any attempts to convert me over ;-) But currently I would prefer a shell script.
Cheers Jan