search

ecmwf / anemoi-datasets

Apache License 2.0
34 stars 19 forks source link

Problems trying to read satellite data from EUMETSAT #18

Open OriolHinojo opened 2 months ago

OriolHinojo commented 2 months ago

What happened?

I want to build an anemoi dataset from EUMETSAT satellite data. My data is in NetCDF format and compliant with the CF conventions. I tried with different datasets but it always crashes due to similar errors that seem to come from formatting.

What are the steps to reproduce the bug?

  1. Download the file attached on the provided link. It corresponds to EO:EUM:DAT:MSG:HRSEVIRI for 2024-08-16T11:45:00 to 2024-08-16T11:57:00. Dataset information
  2. Create recipe.yml file: 
    
dates:
start: 2024-08-16T11:45:00.000000000
end: 2024-08-16T11:57:00.000000000

input: netcdf: path: HRSEVIRI_20240816T114510Z_20240816T115742Z_epct_312df900_F.nc

3. Run "anemoi-datasets create recipe.yaml dataset.zarr".

**Versions**
python 3.10
earthkit 0.6.1
ecmwflibs 0.6.3
anemoi datasets 0.4.4(I also tried with 0.4.5.dev1+g50a)

### Version

 0.4.4 and 0.4.5.dev1+g50a

### Platform (OS and architecture)

Ubuntu 22.04.4 LTS

### Relevant log output

```shell
2024-08-20 15:18:11 INFO Setting flatten_grid=True in config
2024-08-20 15:18:11 INFO Setting ensemble_dimension=2 in config
2024-08-20 15:18:11 INFO Setting flatten_grid=True in config
2024-08-20 15:18:11 INFO Setting ensemble_dimension=2 in config
2024-08-20 15:18:11 INFO {'start': datetime.datetime(2024, 8, 16, 11, 45), 'end': datetime.datetime(2024, 8, 16, 11, 57), 'group_by': 'monthly'}
2024-08-20 15:18:11 INFO Groups(dates=1)
2024-08-20 15:18:11 INFO FunctionAction: path=HRSEVIRI_20240816T114510Z_20240816T115742Z_epct_312df900_F.nc 
2024-08-20 15:18:11 INFO Minimal input (using only the first date) :
2024-08-20 15:18:11 INFO netcdf(['2024-08-16T11:45:00'])
2024-08-20 15:18:11 INFO Config loaded ok:
2024-08-20 15:18:11 INFO Found 1 datetimes.
2024-08-20 15:18:11 INFO Dates: Found 1 datetimes, in 1 groups: 
2024-08-20 15:18:11 INFO Missing dates: 0
2024-08-20 15:18:11 ERROR Error in execute
Traceback (most recent call last):
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/input.py", line 585, in datasource
    return _tidy(self.action.function(FunctionContext(self), self.dates, *args, **kwargs))
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/netcdf.py", line 14, in execute
    return load_many("📁", context, dates, path, *args, **kwargs)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/__init__.py", line 67, in load_many
    result.append(load_one(emoji, context, dates, path, **kwargs))
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/__init__.py", line 51, in load_one
    fs = XarrayFieldList.from_xarray(data, flavour)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/fieldlist.py", line 106, in from_xarray
    grid=guess.grid(coordinates),
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/flavour.py", line 151, in grid
    raise NotImplementedError(f"Expected 1 latitude coordinate, got {len(lat)}")
NotImplementedError: Expected 1 latitude coordinate, got 0
Traceback (most recent call last):
  File "/home/oriol_h/miniconda3/envs/anemoi_env/bin/anemoi-datasets", line 8, in <module>
    sys.exit(main())
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/__main__.py", line 24, in main
    cli_main(__version__, __doc__, COMMANDS)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/utils/cli.py", line 128, in cli_main
    cmd.run(args)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/commands/create.py", line 67, in run
    self.serial_create(args)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/commands/create.py", line 75, in serial_create
    c.create()
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/__init__.py", line 172, in create
    self.init()
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/__init__.py", line 60, in init
    return obj.initialise(check_name=check_name)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/loaders.py", line 289, in initialise
    variables = self.minimal_input.variables
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/input.py", line 479, in variables
    self.build_coords()
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/input.py", line 430, in build_coords
    from_data = self.get_cube().user_coords
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/input.py", line 224, in get_cube
    ds = self.datasource
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/input.py", line 90, in wrapper
    result = method(self, *args, **kwargs)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/template.py", line 26, in wrapper
    result = method(self, *args, **kwargs)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/trace.py", line 56, in wrapper
    result = method(self, *args, **kwargs)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/input.py", line 585, in datasource
    return _tidy(self.action.function(FunctionContext(self), self.dates, *args, **kwargs))
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/netcdf.py", line 14, in execute
    return load_many("📁", context, dates, path, *args, **kwargs)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/__init__.py", line 67, in load_many
    result.append(load_one(emoji, context, dates, path, **kwargs))
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/__init__.py", line 51, in load_one
    fs = XarrayFieldList.from_xarray(data, flavour)
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/fieldlist.py", line 106, in from_xarray
    grid=guess.grid(coordinates),
  File "/home/oriol_h/miniconda3/envs/anemoi_env/lib/python3.10/site-packages/anemoi/datasets/create/functions/sources/xarray/flavour.py", line 151, in grid
    raise NotImplementedError(f"Expected 1 latitude coordinate, got {len(lat)}")
NotImplementedError: Expected 1 latitude coordinate, got 0

Accompanying data

https://wekeo-files.prod.wekeo2.eu/index.php/s/226xtDk4in9sB2p

Organisation

No response

floriankrb commented 2 months ago

The link must have expired, the data is not available anymore

OriolHinojo commented 2 months ago

It did expire. I regenerated the link with the data and changed the settings so it doesn't expire anymore. https://wekeo-files.prod.wekeo2.eu/index.php/s/BspMLpWPoNiWAfT