search

NVIDIA / earth2studio

Open-source deep-learning framework for exploring, building and deploying AI weather/climate workflows.
https://nvidia.github.io/earth2studio/
Apache License 2.0
95 stars 31 forks source link

🚀[FEA]: Package system doesn't natively work with Zarr Stores #115

Open dallasfoster opened 2 months ago

dallasfoster commented 2 months ago

Version

main

On which installation method(s) does this occur?

Pip, Source

Describe the issue

The package .resolve method uses fsspec._open under the hood, which fails if the file in question is interpreted as a directory, as is the case with Zarr stores.

store = package.resolve('metadata.zarr')
package.resolve("metadata.zarr")
  File "/usr/local/lib/python3.10/dist-packages/earth2studio/models/auto.py", line 319, in resolve
    with self.open(file_path) as file:
  File "/usr/local/lib/python3.10/dist-packages/earth2studio/models/auto.py", line 301, in open
    return self.fs.open(full_path, callback=callback)
  File "/usr/local/lib/python3.10/dist-packages/fsspec/spec.py", line 1293, in open
    f = self._open(
  File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 197, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 322, in __init__
    self._open()
  File "/usr/local/lib/python3.10/dist-packages/fsspec/implementations/local.py", line 327, in _open
    self.f = open(self.path, mode=self.mode)
IsADirectoryError: [Errno 21] Is a directory: '/path/to/file/metadata.zarr'
NickGeneva commented 1 month ago

Updating to feature request, doing this on NGC will potentially be quite challenging due to how file access works... specifically for authenticated users.