CSyl
commented
1 month ago Additional Finding: The "ERROR Error in retrieving metadata (cannot build data request info) for XArrayMetadata( ...)." error message also seems to only occur at the initial date at which an input data is set up to be configured at. For example, if I were to set the configuration file as such:
dates: start: 2021-12-31T09:00:00 end: 2021-12-31T22:00:00 frequency: 1h
input: xarray-zarr: url: "gcp_ar_era5_subset.zarr" param: [2m_temperature, 10m_u_component_of_wind, geopotential, 10m_v_component_of_wind, surface_pressure]
The "ERROR Error in retrieving metadata (cannot build data request info) for XArrayMetadata( ...) will only be printed out for all variables at 2021-12-31T09:00:00. If I were to alter that start date, thenthe error will only be printed out for all variables at that new initial date YET the metadata/attributes were in fact verified to be within the input data at those given times via xarray.
b8raoult
mchantry
What happened?
With the latest release of anemoi-dataset, I was able to convert the zarr to an anemoi dataset, however several error messages regarding the metadata retrieval were showing up during the data creation/conversion process of the zarr data. The resulting error would state "ERROR Error in retrieving metadata (cannot build data request info) for XArrayMetadata( ...)."
As a side note, I am able to compute the mean over the grid points of the resulting anemoi datasets, but am concerned about the errors I am getting. Is this an expected error? Could you please assist on why this error may be occurring?
Thank You!
What are the steps to reproduce the bug?
The step-by-step of one of the case scenarios performed for which produces this error:
Data Tested: A subset of ERA5 ZARR from GCP storage saved to local disk was performed via a .py script:
Configuration File Used: A yaml was created with the following content & saved as test_local_gcp_ar_sample_zarr.yaml
Command Executed:
Findings/Results: With the latest release of anemoi-dataset, was able to convert the zarr to an anemoi dataset, however several error messages regarding the metadata retrieval were showing up during the data creation/conversion process of the zarr data. Please see below the "ERROR Error in retrieving metadata (cannot build data request info) for XArrayMetadata( ...)" messages:
2024-09-16 18:06:44 INFO Task init((),{}) starting 2024-09-16 18:06:44 INFO Setting flatten_grid=True in config 2024-09-16 18:06:44 INFO Setting ensemble_dimension=2 in config 2024-09-16 18:06:44 INFO Setting flatten_grid=True in config 2024-09-16 18:06:44 INFO Setting ensemble_dimension=2 in config 2024-09-16 18:06:44 INFO {'start': datetime.datetime(2021, 12, 31, 9, 0), 'end': datetime.datetime(2021, 12, 31, 22, 0), 'frequency': '1h', 'group_by': 'monthly'} 2024-09-16 18:06:44 INFO Groups(dates=1) 2024-09-16 18:06:44 INFO FunctionAction: url=gcp_ar_era5_subset.zarr param=['2m_temperature', '10m_u_component_of_wind', 'geopotential', '10m_v_component_of_wind', 'surface_pressure'] 2024-09-16 18:06:44 INFO Minimal input for 'init' step (using only the first date) : 2024-09-16 18:06:44 INFO xarray-zarr(['2021-12-31T09:00:00']) 2024-09-16 18:06:44 INFO Config loaded ok: 2024-09-16 18:06:44 INFO Found 14 datetimes. 2024-09-16 18:06:44 INFO Dates: Found 14 datetimes, in 1 groups: 2024-09-16 18:06:44 INFO Missing dates: 0
2024-09-16 18:06:44 INFO Found 41 variables : 10m_u_component_of_wind,10m_v_component_of_wind,2m_temperature,geopotential_1,geopotential_10,geopotential_100,geopotential_1000,geopotential_125,geopotential_150,geopotential_175,geopotential_2,geopotential_20,geopotential_200,geopotential_225,geopotential_250,geopotential_3,geopotential_30,geopotential_300,geopotential_350,geopotential_400,geopotential_450,geopotential_5,geopo
ential_50,geopotential_500,geopotential_550,geopotential_600,geopotential_650,geopotential_7,geopotential_70,geopotential_700,geopotential_750,geopotential_775,geopotential_800,geopotential_825,geopotential_850,geopotential_875,geopotential_900,geopotential_925,geopotential_950,geopotential_975,surface_pressure. 2024-09-16 18:06:44 INFO Found 1 ensembles : 0. 2024-09-16 18:06:44 INFO gridpoints size: [65160, 65160] 2024-09-16 18:06:44 INFO resolution=None 2024-09-16 18:06:44 INFO total_shape = [14, 41, 1, 65160] 2024-09-16 18:06:44 INFO chunks=(1, 41, 1, 65160) 2024-09-16 18:06:44 INFO Creating Dataset 'test_local_gcp_ar_sample_zarr.zarr', with total_shape=[14, 41, 1, 65160], chunks=(1, 41, 1, 65160) and dtype='float32' 2024-09-16 18:06:44 ERROR Error in retrieving metadata (cannot build data request info) for XArrayMetadata({'variable': '10m_u_component_of_wind', 'time': '0900', 'date': '20211231', 'step': 0, 'valid_datetime': '2021-12-31T09:00:00'})
Traceback (most recent call last): File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/anemoi/datasets/create/input.py", line 111, in _data_request date = field.datetime()["valid_time"] File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/earthkit/data/core/fieldlist.py", line 512, in datetime return self._metadata.datetime() File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/earthkit/data/core/metadata.py", line 312, in datetime "base_time": self._base_datetime(), File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/metadata.py", line 84, in _base_datetime return self._field.forecast_reference_time File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/field.py", line 106, in forecast_reference_time return self.owner.forecast_reference_time AttributeError: 'Variable' object has no attribute 'forecast_reference_time' 2024-09-16 18:06:44 ERROR Error in retrieving metadata (cannot build data request info) for XArrayMetadata({'variable': '10m_v_component_of_wind', 'time': '0900', 'date': '20211231', 'step': 0, 'valid_datetime': '2021-12-31T09:00:00'})
Traceback (most recent call last): File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/anemoi/datasets/create/input.py", line 111, in _data_request date = field.datetime()["valid_time"] File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/earthkit/data/core/fieldlist.py", line 512, in datetime return self._metadata.datetime() File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/earthkit/data/core/metadata.py", line 312, in datetime "base_time": self._base_datetime(), File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/metadata.py", line 84, in _base_datetime return self._field.forecast_reference_time File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/field.py", line 106, in forecast_reference_time return self.owner.forecast_reference_time AttributeError: 'Variable' object has no attribute 'forecast_reference_time' 2024-09-16 18:06:44 ERROR Error in retrieving metadata (cannot build data request info) for XArrayMetadata({'variable': '2m_temperature', 'time': '0900', 'date': '20211231', 'step': 0, 'valid_datetime': '2021-12-31T09:00:00'})
: : :
AttributeError: 'Variable' object has no attribute 'forecast_reference_time' 2024-09-16 18:06:44 ERROR Error in retrieving metadata (cannot build data request info) for XArrayMetadata({'variable': 'surface_pressure', 'time': '0900', 'date': '20211231', 'step': 0, 'valid_datetime': '2021-12-31T09:00:00'}) Traceback (most recent call last): File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/anemoi/datasets/create/input.py", line 111, in _data_request date = field.datetime()["valid_time"] File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/earthkit/data/core/fieldlist.py", line 512, in datetime return self._metadata.datetime() File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/earthkit/data/core/metadata.py", line 312, in datetime "base_time": self._base_datetime(), File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/metadata.py", line 84, in _base_datetime return self._field.forecast_reference_time File " ~/miniconda3/envs/anemoi_test/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/field.py", line 106, in forecast_reference_time return self.owner.forecast_reference_time AttributeError: 'Variable' object has no attribute 'forecast_reference_time' 2024-09-16 18:06:44 WARNING Dataset name error: the dataset name 'test_local_gcp_ar_sample_zarr' does not follow naming convention. Does not match ^(\w+)-([\w-]+)-(\w+)-(\w+)-(\d\d\d\d)-(\d\d\d\d)-(\d+h)-v(\d+)-?([a-zA-Z0-9-]+)?$ 2024-09-16 18:06:44 INFO Number of years 0 < 10, leaving out 20%. end=np.datetime64('2021-12-31T19:00:00') 2024-09-16 18:06:44 INFO Will compute statistics from 2021-12-31T09:00:00 to 2021-12-31T19:00:00 ❌❌❌ FIXME: Config serialisation to be checked {"attribution": "unknown", "build": {"group_by": "monthly", "use_grib_paramid": false, "variable_naming": "default"}, "config_path": " ~ /anemoi/gh_most_uptodate/test_scripts/test_local_gcp_ar_sample_zarr.yaml", "dataset_status": "experimental", "dates": {"end": "2021-12-31 22:00:00", "frequency": "1h", "group_by": "monthly", "start": "2021-12-31 09:00:00"}, "description": "No description provided.", "input": {"xarray-zarr": {"param": ["2m_temperature", "10m_u_component_of_wind", "geopotential", "10m_v_component_of_wind", "surface_pressure"], "url": "gcp_ar_era5_subset.zarr"}}, "licence": "unknown", "output": {"chunking": {"dates": 1, "ensembles": 1}, "dtype": "float32", "ensemble_dimension": 2, "flatten_grid": true, "order_by": {"number": "ascending", "param_level": "ascending", "valid_datetime": "ascending"}, "remapping": {"paramlevel": "{param}{levelist}"}, "statistics": "param_level"}, "reading_chunks": null, "statistics": {"allow_nans": []}} {"attribution": "unknown", "build": {"group_by": "monthly", "use_grib_paramid": false, "variable_naming": "default"}, "config_path": " ~ /anemoi/gh_most_uptodate/test_scripts/test_local_gcp_ar_sample_zarr.yaml", "dataset_status": "experimental", "dates": {"end": "2021-12-31 22:00:00", "frequency": "1h", "group_by": "monthly", "start": "2021-12-31 09:00:00"}, "description": "No description provided.", "input": {"xarray-zarr": {"param": ["2m_temperature", "10m_u_component_of_wind", "geopotential", "10m_v_component_of_wind", "surface_pressure"], "url": "gcp_ar_era5_subset.zarr"}}, "licence": "unknown", "output": {"chunking": {"dates": 1, "ensembles": 1}, "dtype": "float32", "ensemble_dimension": 2, "flatten_grid": true, "order_by": {"number": "ascending", "param_level": "ascending", "valid_datetime": "ascending"}, "remapping": {"paramlevel": "{param}{levelist}"}, "statistics": "param_level"}, "reading_chunks": null, "statistics": {"allow_nans": []}} 2024-09-16 18:06:44 INFO Task load((),{}) starting 2024-09-16 18:06:44 INFO {'end': '2021-12-31T22:00:00', 'frequency': '1h', 'group_by': 'monthly', 'start': '2021-12-31T09:00:00'} 2024-09-16 18:06:44 INFO Groups(dates=1) 2024-09-16 18:06:44 INFO FunctionAction: param=['2m_temperature', '10m_u_component_of_wind', 'geopotential', '10m_v_component_of_wind', 'surface_pressure'] url=gcp_ar_era5_subset.zarr
Loading 573/574: 100%|████████████████████████████████████████████████| 574/574 [00:29<00:00, 19.78it/s] 2024-09-16 18:07:16 INFO Name : /data Type : zarr.core.Array Data type : float32 Shape : (14, 41, 1, 65160) Chunk shape : (1, 41, 1, 65160) Order : C Read-only : True Compressor : Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0) Store type : zarr.storage.DirectoryStore No. bytes : 149607360 (142.7M) No. bytes stored : 92353145 (88.1M) Storage ratio : 1.6 Chunks initialized : 14/14
2024-09-16 18:07:16 INFO Task finalise((),{}) starting 2024-09-16 18:07:16 INFO Variables minimum maximum mean stdev has_nans 10m_u_component_of_wind -19.22 20.98 -0.19 5.10 0.00 10m_v_component_of_wind -18.40 22.43 -0.31 4.51 0.00 2m_temperature 221.06 316.17 277.19 20.41 0.00 geopotential_1 425099.12 493273.97 465659.10 17453.64 0.00 geopotential_10 275382.28 311927.84 300915.73 9302.10 0.00 geopotential_100 147745.34 163643.91 157286.27 5051.06 0.00 geopotential_1000 -3148.51 3412.69 803.08 1071.14 0.00 geopotential_125 134741.56 151257.64 143845.12 5458.15 0.00 geopotential_150 123852.06 140592.05 132735.76 5688.41 0.00 geopotential_175 114547.15 131239.23 123239.62 5779.90 0.00 geopotential_2 375595.91 435469.00 413513.09 15086.42 0.00 geopotential_20 237253.33 263741.97 256186.92 6838.97 0.00 geopotential_200 106404.47 122853.99 114930.94 5763.09 0.00 geopotential_225 99154.48 115186.59 107532.01 5657.73 0.00 geopotential_250 92567.17 108154.01 100847.66 5482.72 0.00 geopotential_3 348050.31 402489.88 383738.78 13593.41 0.00 geopotential_30 215299.84 236499.61 230671.61 5568.11 0.00 geopotential_300 81098.05 95595.55 89073.25 5014.69 0.00 geopotential_350 71283.67 84574.07 78837.49 4508.30 0.00 geopotential_400 62706.44 74810.72 69719.14 4029.09 0.00 geopotential_450 55092.68 66061.89 61473.06 3593.01 0.00 geopotential_5 315959.47 362728.69 347604.26 11759.93 0.00 geopotential_50 187061.36 204328.94 199091.86 4535.69 0.00 geopotential_500 48186.63 58045.56 53934.92 3200.09 0.00 geopotential_550 41836.48 50625.60 46985.87 2844.82 0.00 geopotential_600 35929.71 43804.48 40536.81 2522.16 0.00 geopotential_650 30078.14 37560.64 34516.64 2227.84 0.00 geopotential_7 295866.34 337658.84 324624.08 10574.69 0.00 geopotential_70 168151.08 183816.19 178658.50 4480.88 0.00 geopotential_700 24547.81 31691.87 28867.42 1960.86 0.00 geopotential_750 19335.88 26146.02 23545.67 1721.18 0.00 geopotential_775 16833.86 23534.42 20996.56 1611.84 0.00 geopotential_800 14402.37 21070.07 18516.78 1509.99 0.00 geopotential_825 12036.17 18667.33 16103.02 1416.23 0.00 geopotential_850 9733.97 16322.85 13752.33 1331.52 0.00 geopotential_875 7452.32 14033.64 11461.61 1256.85 0.00 geopotential_900 5226.78 11806.53 9227.80 1193.17 0.00 geopotential_925 3055.42 9639.81 7047.96 1141.47 0.00 geopotential_950 938.20 7520.03 4918.94 1102.91 0.00 geopotential_975 -1126.84 5445.05 2838.09 1079.21 0.00 surface_pressure 51776.86 104166.40 96732.04 9546.72 0.00
2024-09-16 18:07:16 INFO Wrote statistics in test_local_gcp_ar_sample_zarr.zarr
Computing size of test_local_gcp_ar_sample_zarr.zarr: 16it [00:00, 32186.51it/s]
2024-09-16 18:07:16 INFO Total size: 88.1 MiB 2024-09-16 18:07:16 INFO Total number of files: 60 2024-09-16 18:07:16 INFO Task patch((),{}) starting 2024-09-16 18:07:16 INFO ✅ Remove _create_yaml_config 2024-09-16 18:07:16 INFO Dataset changed by patch 2024-09-16 18:07:16 INFO Task init_additions((),{}) starting 2024-09-16 18:07:16 WARNING No delta found in kwargs, no addtions will be computed. 2024-09-16 18:07:16 INFO Task run_additions((),{}) starting 2024-09-16 18:07:16 WARNING No delta found in kwargs, no addtions will be computed. 2024-09-16 18:07:16 INFO Task finalise_additions((),{}) starting 2024-09-16 18:07:16 WARNING No delta found in kwargs, no addtions will be computed.
Computing size of test_local_gcp_ar_sample_zarr.zarr: 16it [00:00, 32832.13it/s]
2024-09-16 18:07:16 INFO Total size: 88.1 MiB 2024-09-16 18:07:16 INFO Total number of files: 60 2024-09-16 18:07:16 INFO Task cleanup((),{}) starting 2024-09-16 18:07:16 INFO Task verify((),{}) starting 2024-09-16 18:07:16 INFO Verifying dataset at test_local_gcp_ar_sample_zarr.zarr 2024-09-16 18:07:16 INFO test_local_gcp_ar_sample_zarr.zarr 2024-09-16 18:07:16 INFO Create completed in 32 seconds
Also, I was getting these similar metadata errors when I tried to create an anemoi dataset against a netcdf file stored in S3 that the current anemoi-framework is referencing within one of its scripts (e.g. s3://nsf-ncar-era5/e5.oper.an.pl/202403/e5.oper.an.pl.128_060_pv.ll025sc.2024030100_2024030123.nc), but in this scenario I created the json for that nc file and referenced it in a kerchunk-xarray yaml config file as such:
Creating the JSON for the NetCDF
Configuration File Used (xarray-kerchunk yaml formatted per anemoi guide)
Version
0.4.5
Platform (OS and architecture)
Linux
Relevant log output
No response
Accompanying data
gs://gcp-public-data-arco-era5/ar/1959-2022-1h-360x181_equiangular_with_poles_conservative.zarr, s3://nsf-ncar-era5/e5.oper.an.pl/202403/"e5.oper.an.pl.128_060_pv.ll025sc.2024030100_2024030123.nc
Organisation
No response