search

google-research / weatherbench2

A benchmark for the next generation of data-driven global weather models.
https://weatherbench2.readthedocs.io
Apache License 2.0
368 stars 33 forks source link

Evaluate.py issues, '.zmetadata' not found #127

Open jdwillard19 opened 4 months ago

jdwillard19 commented 4 months ago

Issues running evaluate.py

Description

I tried to run one of the examples from the "official evaluation" section of the weatherbench site. But, it doesn't seem to run and gives some backend errors. This seems to happen on all the scripts I tried to run from that page.

Steps to Reproduce

  1. I ran 
    python evaluate.py -- \
--forecast_path=gs://weatherbench2/datasets/ifs_ens/2018-2022-1440x721.zarr \
--obs_path=gs://weatherbench2/datasets/era5/1959-2022-6h-1440x721.zarr \
--climatology_path=gs://weatherbench2/datasets/era5-hourly-climatology/1990-2019_6h_1440x721.zarr \
--output_dir=/pscratch/sd/j/jwillard/FCN_exp/wb_test_eval/ \
--output_file_prefix=ifs_ens_vs_era5_2018_ \
--input_chunks=init_time=1,lead_time=1 \
--fanout=27 \
--regions=all \
--eval_configs=probabilistic \
--evaluate_climatology=False \
--evaluate_persistence=False \
--time_start=2018-01-01 \
--time_stop=2018-12-31 \
--variables=temperature\
--add_land_region=False \
--use_beam=True

Result

I get the following stack trace

Traceback (most recent call last):
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/xarray/backends/zarr.py", line 427, in open_group
    zarr_group = zarr.open_consolidated(store, **open_kwargs)
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/zarr/convenience.py", line 1351, in open_consolidated
    meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/zarr/storage.py", line 2947, in __init__
    meta = json_loads(self.store[metadata_key])
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/zarr/storage.py", line 764, in __getitem__
    return self._mutable_mapping[key]
KeyError: '.zmetadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/xarray/backends/zarr.py", line 430, in open_group
    zarr_group = zarr.open_group(store, **open_kwargs)
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/zarr/hierarchy.py", line 1525, in open_group
    raise GroupNotFoundError(path)
zarr.errors.GroupNotFoundError: group not found at path ''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/global/u2/j/jwillard/weatherbench2/scripts/evaluate.py", line 605, in <module>
    app.run(main)
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/global/u2/j/jwillard/weatherbench2/scripts/evaluate.py", line 354, in main
    land_sea_mask = xr.open_zarr(OBS_PATH.value)['land_sea_mask'].compute()
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/xarray/backends/zarr.py", line 867, in open_zarr
    ds = open_dataset(
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/xarray/backends/api.py", line 570, in open_dataset
    backend_ds = backend.open_dataset(
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/xarray/backends/zarr.py", line 934, in open_dataset
    store = ZarrStore.open_group(
  File "/global/cfs/cdirs/m4416/jared/fcn_mip-env_e2mip_update/lib/python3.10/site-packages/xarray/backends/zarr.py", line 447, in open_group
    raise FileNotFoundError(f"No such file or directory: '{store}'")
FileNotFoundError: No such file or directory: 'None'

Environment

raspstephan commented 3 months ago

Ah sorry for the late reply but if this is still an issue, could you run again. It might have been because I moved some files around.

bonevbs commented 3 months ago

@raspstephan I am seeing the same issue as well. My understanding is that it is looking for the consolidated metadata file - from what I see it should exist in the above files on gs.

raspstephan commented 2 months ago

Ok, I will look into it thanks!

SadeghTabas-NOAA commented 2 months ago

@raspstephan I have the same issue as well.

raspstephan commented 2 months ago

I think it might just be an issue with an extra -- in the command. Does it help when you try:

python evaluate.py \
  --forecast_path=gs://weatherbench2/datasets/ifs_ens/2018-2022-1440x721.zarr \
...