Open jbusecke opened 3 months ago
Interestingly, I seem to have different engine-error combinations. However, I am trying to load directly from the url without any Pangeo Forge wrappers (direct from hub notebook)
with fsspec.open(url, mode='rb').open() as file: # url is 'https://huggingface.co/.../ClimSim_low-res...E3SM-MMF.mli.0002-01-01-00000.nc' xr.open_dataset(file, chunks={}, use_cftime=True)
or specifying engine='scipy'
yields the same
IndexError: index 0 is out of bounds for axis 0 with size 0
'engine= h5netcdf'
gives me
ValueError: b'CDF\x05\x00\x00\x00\x00' is not the signature of a valid netCDF4 file
engine="netcdf4" yields
ValueError: can only read bytes or file-like objects with engine='scipy' or 'h5netcdf'
I played around with huggingface datasets library but was bottlenecked by the fact that it can't parse netcdf files (.nc is not a supported file extension)
So I threw up my hands, used requests to make a get requests and manually read the bytes.
resp = requests.get(url) # status code 200
resp.content # b'CDF\x05\x00\x00\x00\x00 ... '
xr.open_dataset(resp.content) # IndexError: index 0 is out of bounds for axis 0 with size 0
xr.open_dataset(io.BytesIO(resp.content)) # also gives IndexError (io.BytesIO(response.content) by itself throws no error)
with open("file.nc", 'wb') as f:
f.write(resp.content)
xr.open_dataset("file.nc") # works??
So, I guess this echoes copy_to_local being necessary for some strange unnecessary reason. But to me, this indicates that it is not a ffspec error but either a error with the files or an error with how xarray processes raw string binary data versus file pointers?
*using requests was not necessary. ffspec.open().read() gives me the same byte string that I can save and use
I just confirmed that the newly uploaded 'expanded' data (https://huggingface.co/datasets/LEAP/ClimSim_low-res-expanded/tree/main/train/0001-02) (thanks to @zyhu-hu !) does not suffer from these issues. I will try to target these for a virtual zarr dataset generation https://github.com/leap-stc/climsim_feedstock/issues/9, but they might also serve as a good comparison case here.
So I stumbled upon Charles' code here and wanted to formalize this a bit more.
I would love to get rid of the copy_to_local but there seems to be currently no way!
So i took a look at some of the cached files:
gives
I can reproduce this locally by using
engine='scipy'
, but ifengine='netcdf4'
(the default) it just works. This is super weird.Is there something broken with fsspec? Or with the files themselves?