search

ESMValGroup / ESMValCore

ESMValCore: A community tool for pre-processing data from Earth system models in CMIP and running analysis scripts.
https://www.esmvaltool.org
Apache License 2.0
42 stars 38 forks source link

Add `esmvaltool compare` command to run `esmvaltool/utils/testing/regression/compare.py` #2381

Open alistairsellar opened 5 months ago

alistairsellar commented 5 months ago

Is your feature request related to a problem? Please describe. As discussed in https://github.com/ESMValGroup/ESMValTool/issues/3437, it would be useful for recipe test workflow (and more generally for users) to have an esmvaltool compare command that uses the functions in esmvaltool/utils/testing/regression/compare.py to compare results from two recipe runs.

Would you be able to help out? Yes, I would make this change.

alistairsellar commented 5 months ago

Having scoped the code, I propose that the command (at least in first implementation) compares a single "current" run with the reference, in contrast to compare.py which allows for a list of "current" runs to each be compared to the reference. This approach would allow the command to call the existing compare() function without changes to ESMValTool, whereas a list would require refactoring of main(), or duplication of some of its code into Core. Users who need the more advanced multi-run comparison in a single call can run compare.py itself.

Tagging @bouweandela for interest and opportunity to disagree with proposal.

bouweandela commented 5 months ago

Nice idea! You may want to consider if this will be the only recipe development and testing utility that you want to integrate, or if there might be more in the future. In the latter case, it might be nicer to group them under e.g. an esmvaltool development command, to avoid that we end up with 25 esmvaltool something commands.

alistairsellar commented 5 months ago

Good point @bouweandela, will have a think about that.

alistairsellar commented 5 months ago

Nice idea! You may want to consider if this will be the only recipe development and testing utility that you want to integrate, or if there might be more in the future. In the latter case, it might be nicer to group them under e.g. an esmvaltool development command, to avoid that we end up with 25 esmvaltool something commands.

Thanks @bouweandela, have thought about this and discussed with @ehogan. We can't think of any other development commands, apart from potentially esmvaltool config validate (discussed in ESMValGroup/ESMValTool#3392) which would be in the existing config command group.

So I'll get back to my draft PR for an esmvaltool compare command.