search

ESMValGroup / ESMValTool

ESMValTool: A community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
https://www.esmvaltool.org
Apache License 2.0
217 stars 127 forks source link

Recipe perfmetrics_CMIP5: ta_Amon_EC-EARTH_historical_r6i1p1 unavailable on ESGF #3303

Open rbeucher opened 1 year ago

rbeucher commented 1 year ago

Hi All,

I am running the perfmetrics_CMIP5 recipe and the ESGF search cannot find the dataset ta_Amon_EC-EARTH_historical_r6i1p1*.nc on ESGF

2023-07-21 01:59:22,878 UTC [2484833] DEBUG   Initialized SQLiteDict with serializer: None
2023-07-21 01:59:22,879 UTC [2484833] DEBUG   Opening connection to /home/563/rb5533/.esmvaltool/cache/pyesgf-search-results.sqlite:redirects
2023-07-21 01:59:22,885 UTC [2484833] DEBUG   Cache directives from request headers: CacheDirectives()
2023-07-21 01:59:22,885 UTC [2484833] DEBUG   Pre-read cache checks: Passed
2023-07-21 01:59:22,888 UTC [2484833] DEBUG   Post-read cache actions: CacheActions(expire_after=86400)
2023-07-21 01:59:22,888 UTC [2484833] DEBUG   Closing backend connections
2023-07-21 01:59:22,889 UTC [2484833] DEBUG   Found the following files matching facets {'project': 'CMIP5', 'model': 'EC-EARTH', 'ensemble': 'r6i1p1', 'experiment': 'historical', 'time_frequency': 'mon', 'institute': 'ICHEC', 'cmor_table': 'Amon', 'product': 'output1,output2', 'variable': 'ta'}: none
2023-07-21 01:59:22,889 UTC [2484833] DEBUG   Selected files:

2023-07-21 01:59:22,889 UTC [2484833] ERROR   No input files found for Dataset:
{'diagnostic': 'taZONAL',
 'variable_group': 'ta',
 'dataset': 'EC-EARTH',
 'project': 'CMIP5',
 'mip': 'Amon',
 'short_name': 'ta',
 'alias': 'CMIP5_EC-EARTH',
 'alternative_dataset': 'NCEP-NCAR-R1',
 'ensemble': 'r6i1p1',
 'exp': 'historical',
 'frequency': 'mon',
 'institute': ['ICHEC'],
 'long_name': 'Air Temperature',
 'modeling_realm': ['atmos'],
 'original_short_name': 'ta',
 'preprocessor': 'ppALL',
 'product': ['output1', 'output2'],
 'recipe_dataset_index': 19,
 'reference_dataset': 'ERA-Interim',
 'standard_name': 'air_temperature',
 'timerange': '2000/2002',
 'units': 'K'}
session: 'recipe_perfmetrics_CMIP5_20230721_015800'
2023-07-21 01:59:22,889 UTC [2484833] ERROR   Looked for files matching
/g/data/al33/replicas/CMIP5/combined/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r6i1p1/*/ta/ta_Amon_EC-EARTH_historical_r6i1p1*.nc
/g/data/kj13/datasets/esmvaltool/esmvaltool_climate_data/cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r6i1p1/*/ta/ta_Amon_EC-EARTH_historical_r6i1p1*.nc
/g/data/r87/DRSv3/CMIP5/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r6i1p1/*/ta/ta_Amon_EC-EARTH_historical_r6i1p1*.nc
/g/data/rr3/publications/CMIP5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r6i1p1/*/ta/ta_Amon_EC-EARTH_historical_r6i1p1*.nc

I don't have that dataset available at NCI.

I checked ESGF and that is all I could find:

image

rbeucher commented 1 year ago

This also affects perfmetrics_CMIP5_4cds

- Missing data for Dataset: ta, Amon, CMIP5, CESM1-CAM5-1-FV2, historical, r1i1p1
- Missing data for Dataset: ta, Amon, CMIP5, EC-EARTH, historical, r6i1p1
2023-07-21 05:30:04,373 UTC [2876485] ERROR   Missing data for preprocessor ua850/ua:
- Missing data for Dataset: ua, Amon, CMIP5, CESM1-CAM5-1-FV2, historical, r1i1p1
- Missing data for Dataset: ua, Amon, CMIP5, EC-EARTH, historical, r6i1p1
2023-07-21 05:30:04,373 UTC [2876485] ERROR   Missing data for preprocessor ua200/ua:
- Missing data for Dataset: ua, Amon, CMIP5, CESM1-CAM5-1-FV2, historical, r1i1p1
- Missing data for Dataset: ua, Amon, CMIP5, EC-EARTH, historical, r6i1p1
2023-07-21 05:30:04,373 UTC [2876485] ERROR   Missing data for preprocessor zg500/zg:
- Missing data for Dataset: zg, Amon, CMIP5, CESM1-CAM5-1-FV2, historical, r1i1p1
2023-07-21 05:30:04,373 UTC [2876485] ERROR   Missing data for preprocessor hus400/hus:
- Missing data for Dataset: hus, Amon, CMIP5, CESM1-CAM5-1-FV2, historical, r1i1p1
2023-07-21 05:30:04,373 UTC [2876485] ERROR   Missing data for preprocessor ts/ts:
- Missing data for Dataset: ts, Amon, CMIP5, CESM1-CAM5-1-FV2, historical, r1i1p1
2023-07-21 05:30:04,373 UTC [2876485] ERROR   Missing data for preprocessor pr/pr:
- Missing data for Dataset: pr, Amon, CMIP5, CESM1-CAM5-1-FV2, historical, r1i1p1
2023-07-21 05:30:04,373 UTC [2876485] ERROR   Missing data for preprocessor clt/clt:
- Missing data for Dataset: clt, Amon, CMIP5, CESM1-CAM5-1-FV2, historical, r1i1p1
2023-07-21 05:30:04,373 UTC [2876485] ERROR   Missing data for preprocessor rsut/rsut:
- Missing data for Dataset: rsut, Amon, CMIP5, CESM1-CAM5-1-FV2, historical, r1i1p1
valeriupredoi commented 1 year ago

I can conform that BADC on CEDA-JASMIN don't have ta in /badc/cmip5/data/cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r6i1p1/latest/ but they have it from 1850 to 2005 in v20130108 so try setting the the node to CEDA and pop the version too? I don't know why this is the case, but then again, CMIP5 is becaoming so ancient, places like DKRZ have already started permanently removing it :disappointed:

valeriupredoi commented 1 year ago

in fact, @remi-kazeroni and @schlunma have had to assemble a pseudo-personal cache of CMIP5 data on DKRZ for exactly this reason - data disappearing over the Styx :grin:

remi-kazeroni commented 1 year ago

In this case, the data is still available in the local pool of CMIP5 data at DKRZ: /work/kd0956/CMIP5/data/cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r6i1p1/v20130315/ta/ta_Amon_EC-EARTH_historical_r6i1p1_195001-200512.nc which was used when testing the recipe for the last release. There is not much we can do here because DKRZ is not really maintaining their local replica of CMIP5 data.

The data is indeed unavailable on ESGF:

find_files(project='CMIP5', short_name=['ta'], dataset='EC-EARTH', exp='historical', mip='Amon', timerange='2000/2004')
[ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r12i1p1/v20131231/ta_Amon_EC-EARTH_historical_r12i1p1_200001-201212.nc on hosts ['esgf.ceda.ac.uk'],
 ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r1i1p1/v20131231/ta_Amon_EC-EARTH_historical_r1i1p1_200001-200012.nc on hosts ['esgf.ceda.ac.uk', 'esgf.ceda.ac.uk'],
 ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r1i1p1/v20131231/ta_Amon_EC-EARTH_historical_r1i1p1_200101-200112.nc on hosts ['esgf.ceda.ac.uk', 'esgf.ceda.ac.uk'],
 ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r1i1p1/v20131231/ta_Amon_EC-EARTH_historical_r1i1p1_200201-200212.nc on hosts ['esgf.ceda.ac.uk', 'esgf.ceda.ac.uk'],
 ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r1i1p1/v20131231/ta_Amon_EC-EARTH_historical_r1i1p1_200301-200312.nc on hosts ['esgf.ceda.ac.uk', 'esgf.ceda.ac.uk'],
 ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r1i1p1/v20131231/ta_Amon_EC-EARTH_historical_r1i1p1_200401-200412.nc on hosts ['esgf.ceda.ac.uk', 'esgf.ceda.ac.uk'],
 ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r2i1p1/v20130404/ta_Amon_EC-EARTH_historical_r2i1p1_200001-201212.nc on hosts ['aims3.llnl.gov', 'esgf.ceda.ac.uk', 'esgf2.dkrz.de'],
 ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r7i1p1/v20130212/ta_Amon_EC-EARTH_historical_r7i1p1_200001-201212.nc on hosts ['esgf.ceda.ac.uk', 'esgf2.dkrz.de'],
 ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r8i1p1/v20120711/ta_Amon_EC-EARTH_historical_r8i1p1_195001-201212.nc on hosts ['esgf.ceda.ac.uk', 'esgf2.dkrz.de'],
 ESGFFile:cmip5/output1/ICHEC/EC-EARTH/historical/mon/atmos/Amon/r9i1p1/v20130315/ta_Amon_EC-EARTH_historical_r9i1p1_200001-201212.nc on hosts ['aims3.llnl.gov', 'esgf.ceda.ac.uk', 'esgf2.dkrz.de']]

I did a quick search and the ensembles r7i1p1, r8i1p1, r9i1p1 seem to contain all variables needed. @rbeucher, could you maybe try to rerun the affected recipes with one of these ensembles? If the runs are successful, you could then open a PR and we would ask someone more familiar than me with these recipes to review it.