Add `esmvaltool compare` command to compare the results of two recipe runs

ESMValGroup / ESMValCore

ESMValCore: A community tool for pre-processing data from Earth system models in CMIP and running analysis scripts.

https://www.esmvaltool.org

Apache License 2.0

42 stars 38 forks source link

Add `esmvaltool compare` command to compare the results of two recipe runs #2383

Open alistairsellar opened 5 months ago

alistairsellar commented 5 months ago

Description

Add an esmvaltool compare command to compare the results of two recipes runs, exposing the functionality of esmvaltool/utils/testing/regression/compare.py on the command line.

Closes #2381

Link to documentation: TBA

Before you get started

[x] ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[x] 🧪 The new functionality is relevant and scientifically sound
[x] 🛠 This pull request has a descriptive title and labels
[x] 🛠 Code is written according to the code quality guidelines
[ ] 🧪 and 🛠 Documentation is available
[ ] 🛠 Unit tests have been added
[x] 🛠 Changes are backward compatible
[x] 🛠 Any changed dependencies have been added or removed correctly
[x] 🛠 The list of authors is up to date
[ ] 🛠 All checks below this pull request were successful

alistairsellar commented 4 months ago

Hi @bouweandela, here is a draft of how the new esmvaltool compare command could work. Early comments welcome.

It runs successfully for me, in an esmvaltool environment. However, GitHub tests fail, I think because esmvaltool is not available for esmvalcore testing (E ModuleNotFoundError: No module named 'esmvaltool'). This would make sense, because you want to be able to test esmvalcore standalone without a dependency on esmvaltool.

If I've understood that correctly, would you have any concerns about me moving compare.py from the esmvaltool repository to esmvalcore?

Also, I'd welcome your opinion on what tests should be added for a new command, and where/whether documentation should be updated. On the second question, as far as I can tell, some commands are documented in the "getting started" section. This compare command could be described at the end of https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/output.html?

bouweandela commented 4 months ago

It runs successfully for me, in an esmvaltool environment. However, GitHub tests fail, I think because esmvaltool is not available for esmvalcore testing (E ModuleNotFoundError: No module named 'esmvaltool'). This would make sense, because you want to be able to test esmvalcore standalone without a dependency on esmvaltool.

The easiest solution would be to just add it here instead of in this repository: https://github.com/ESMValGroup/ESMValTool/blob/b8a7b364d141c404a49cae33e7aeabb5fbf7cc36/setup.py#L255

Or are you keen on including this in ESMValCore?

alistairsellar commented 4 months ago

The easiest solution would be to just add it here instead of in this repository: https://github.com/ESMValGroup/ESMValTool/blob/b8a7b364d141c404a49cae33e7aeabb5fbf7cc36/setup.py#L255

Ahah! Thanks @bouweandela - I'll do that.