Open schlunma opened 2 weeks ago
Whoops, linked the wrong recipe; the one above will not lead to this error. Will update.
Updated it. It looks like you need multiple fx variables and multiple datasets to reproduce this. This is not an issue in v2.11.0, so we don't need to include a possible fix into the release branch.
wait - why not an issue in 2.11? Does this manifest itself after the 2.11 branch has been created? :beer:
The problem only appears with the new "DRS per rootpath" feature of v2.12. I couldn't reproduce it in v2.11 (though that does not mean it's not there).
The problem basically boils down to this: the code tries to sort the following object when creating the filled recipe using sorted
:
[(('ensemble', 'r0i0p0'), ('institute', 'IPSL'), ('mip', 'fx'), ('product', ('output1', 'output2')), ('short_name', 'sftof')), (('ensemble', 'r0i0p0'), ('institute', 'IPSL'), ('mip', 'fx'), ('product', 'output1'), ('short_name', 'areacello'))]
The problem here is that the first element contains ('product', ('output1', 'output2'))
and the second one ('product', 'output1')
. Comparing those two fails with this TypeError: '<' not supported between instances of 'str' and 'tuple'
.
The two entries for product
come from this file. I have no idea, however, why for one dataset product is output1
and for the other it's ('output1', 'output2')
. There is no data available in /work/bd0854/DATA/ESMValTool2/download/cmip5/output2
, so data availability does not get taken into account (I think).
I am not quite sure how to proceed. The obvious way of fixing this is via using a key
for sorted
, but since the input to this sorted
call can be arbitrarily nested this is not at all trivial.
The much easier solution is to drop sorted
. This works fine (I also tested the produced filled recipe).
@ESMValGroup/technical-lead-development-team opinions?
ah modern problems - 2.11dev, was a bit confused by the versioning :grin: Thanks, Manu :beer:
All right, after playing around with this for an hour this problem miraculously solved itself 🤯
At some point the tool downloaded some files, and after that it worked out fine. Hopefully this was just a weird curiosity resulting from a weird state of my local data repository. Closing this for now...
Ah, I think I finally understand what's happening there: after using this rootpath
rootpath:
CMIP5:
/work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ: DKRZ
/work/bd0854/DATA/ESMValTool2/download: ESGF
with deleting some downloaded files the tool eventually tries to get the files for the IPSL dataset from the following sources:
2024-07-02 15:41:13,359 UTC [581496] DEBUG esmvalcore._recipe.recipe:270 Using input files for variable fgco2 of dataset IPSL-CM5A-LR:
/work/bd0854/DATA/ESMValTool2/download/cmip5/output1/IPSL/IPSL-CM5A-LR/esmHistorical/mon/ocnBgchem/Omon/r1i1p1/v20120430/fgco2_Omon_IPSL-CM5A-LR_esmHistorical_r1i1p1_185001-200512.nc
with files for supplementary variable sftof:
/work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ/IPSL/IPSL-CM5A-LR/esmHistorical/fx/ocean/fx/r0i0p0/v20120430/sftof/sftof_fx_IPSL-CM5A-LR_esmHistorical_r0i0p0.nc
with files for supplementary variable areacello:
/work/bd0854/DATA/ESMValTool2/download/cmip5/output1/IPSL/IPSL-CM5A-LR/esmHistorical/fx/ocean/fx/r0i0p0/v20120430/areacello_fx_IPSL-CM5A-LR_esmHistorical_r0i0p0.nc
You can see that the two fx files come from the two different rootpaths. Since only one of them provides the product
facet in the DRS
CMIP5:
input_dir:
DKRZ: '{institute}/{dataset}/{exp}/{frequency}/{modeling_realm}/{mip}/{ensemble}/{version}/{short_name}'
ESGF: '{project.lower}/{product}/{institute}/{dataset}/{exp}/{frequency}/{modeling_realm}/{mip}/{ensemble}/{version}'
...
it will be replaced by the proper value for ESGF (areacello), but stays the original tuple (output1, output2) for DKRZ (sftof).
Describe the bug
The recipe
when ran with this configuration
fails with the following error:
See also https://github.com/ESMValGroup/ESMValTool/issues/3693#issuecomment-2194410283.