search

coecms / era5grib

Convert NCI ERA5 archive data to GRIB format
Apache License 2.0
6 stars 1 forks source link

save_grib fails when era5land option True #11

Open paolap opened 2 years ago

paolap commented 2 years ago

When running both era5_um and era5_wrf with eraland, fails when saving data to intermediary file with climtas.to_netcdf_throttled()

Example

era5grib um  --time 198106010000 --output erailand.grib --target="/scratch/e14/cp3790/cylc-run/alt-1981-constant-forcing/share/data/ancils/tasman/ERA/qrparm.mask"

Error

INFO:root:Time: Target (198106010000)
/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.01/lib/python3.9/site-packages/mule/stashmaster.py:284: UserWarning: Ancillary files do not define the UM version number in the Fixed Length Header. No STASHmaster file loaded: Fields will not have STASH entries attached.
  warnings.warn(msg)
INFO:root:Latitudes: Target (-56.75:-22.25)
INFO:root:Longitudes: Target (130.25:194.75)
INFO:root:Creating intermediate file
Traceback (most recent call last):
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.01/bin/era5grib", line 10, in <module>
    sys.exit(main())
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.01/lib/python3.9/site-packages/era5grib/era5grib.py", line 341, in main
    func(**dargs)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.01/lib/python3.9/site-packages/era5grib/era5grib.py", line 238, in era5grib_um
    save_grib(ds, output, format=format)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.01/lib/python3.9/site-packages/era5grib/era5grib.py", line 55, in save_grib
    climtas.io.to_netcdf_throttled(ds, tmp_compressed)
  File "/home/581/pxp581/.local/lib/python3.9/site-packages/climtas-0.3.2+3.g0a4e8f1.dirty-py3.9.egg/climtas/io.py", line 132, in to_netcdf_throttled
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-22.01/lib/python3.9/site-packages/numpy/lib/mixins.py", line 21, in func
    return ufunc(self, other)
TypeError: operand type(s) all returned NotImplemented from __array_ufunc__(<ufunc 'equal'>, '__call__', <COO: shape=(6483600,), dtype=float32, nnz=8, fill_value=0.0>, <function store_chunk at 0x14c6b785b160>): 'COO', 'function'

and the same happens with different environments and WRF.

While running the same with: eragrib um/wrf --no-era5land .... works.

paolap commented 2 years ago

@ScottWales I can see this issue happens when writing the old_graph tasks to the new_graph with a dataset generated by xesmf.Regridder. I'm also assuming it has something to do with the issue you flagged with them

https://github.com/pangeo-data/xesmf/issues/127

However, their solution doesn't seem to work. Any idea of how to move this forward? For the time being we have put together a workaround that simply avoids using to_netcdf_throttled() when saving a UM file, as these are quite small that appears to work, but the WRF files are quite big, I haven't yet tested if just using to_netcdf() might be sufficient there.

Just generally if these repository and climtas are used by the wider ACCESS community we need to discuss as some stage who is responsible for them.

paolap commented 1 year ago

I've apply the temporary fix bypassing to_netcdf_throttled to all save_grib calls with era5land True, so removed model argument and added era5land bool instead