Can't .compute() .zarr anymore. How could I download ERA5 .netcdf data first, and then change to .zarr format for neuralgcm input?

[weatherforecasterwhai]

Hi, I can't .compute() .zarr like months ago. It even can't compute() out in one night. Like #112, and https://github.com/google-research/arco-era5/issues/67.

So, I try to download ERA5 .netcdf data first, and then change to .zarr format for neuralgcm input. However, the format always send errors. 1.Do you have better suggestions for changing .netcdf to .zarr for neuralgcm input? 2.Since .netcdf is so popular in weather forecasting area. Will you make a .netcdf input demo in the future? Or, please make the .compute() easier like before. Thank you!

[weatherforecasterwhai]

I tried several times, only 1G data, for most of times, .compute() is really slow, but with no errors.

Today, I got this error again and again. Would you please help me with this: Traceback (most recent call last): File "/share/home/njhpc/Wanghai/neuralgcm0805/neuralgcm_new.py", line 80, in full_era5_ds = xarray.open_zarr(era5_path, chunks=None) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/backends/zarr.py", line 1103, in open_zarr ds = open_dataset( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/backends/api.py", line 611, in open_dataset backend_ds = backend.open_dataset( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/backends/zarr.py", line 1188, in open_dataset ds = store_entrypoint.open_dataset( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/backends/store.py", line 58, in open_dataset ds = Dataset(vars, attrs=attrs) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/dataset.py", line 718, in init variables, coordnames, dims, indexes, = merge_data_and_coords( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/dataset.py", line 432, in merge_data_and_coords return merge_core( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/merge.py", line 699, in merge_core collected = collect_variables_and_indexes(aligned, indexes=indexes) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/merge.py", line 362, in collect_variables_and_indexes idx, idx_vars = create_default_index_implicit(variable) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/indexes.py", line 1420, in create_default_index_implicit index = PandasIndex.from_variables(dim_var, options={}) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/indexes.py", line 654, in from_variables obj = cls(data, dim, coord_dtype=var.dtype) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/indexes.py", line 589, in init index = safe_cast_to_index(array) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/indexes.py", line 469, in safe_cast_to_index index = pd.Index(np.asarray(array), kwargs) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/indexing.py", line 526, in array return np.asarray(self.get_duck_array(), dtype=dtype) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/coding/variables.py", line 81, in get_duck_array return self.func(self.array.get_duck_array()) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/indexing.py", line 651, in get_duck_array array = self.array[self.key] File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/backends/zarr.py", line 104, in getitem return indexing.explicit_indexing_adapter( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/core/indexing.py", line 1015, in explicit_indexing_adapter result = raw_indexing_method(raw_key.tuple) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/xarray/backends/zarr.py", line 94, in _getitem return self._array[key] File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/zarr/core.py", line 797, in getitem result = self.get_basic_selection(pure_selection, fields=fields) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/zarr/core.py", line 923, in get_basic_selection return self._get_basic_selection_nd(selection=selection, out=out, fields=fields) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/zarr/core.py", line 965, in _get_basic_selection_nd return self._get_selection(indexer=indexer, out=out, fields=fields) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/zarr/core.py", line 1340, in _get_selection self._chunk_getitems( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/zarr/core.py", line 2181, in _chunk_getitems cdatas = self.chunk_store.getitems(ckeys, contexts=contexts) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/zarr/storage.py", line 1435, in getitems raise v File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/fsspec/asyn.py", line 245, in _run_coro return await asyncio.wait_for(coro, timeout=timeout), i File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/gcsfs/core.py", line 1062, in _cat_file headers, out = await self._call("GET", u2, headers=head) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/gcsfs/core.py", line 447, in _call status, headers, info, contents = await self._request( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/decorator.py", line 221, in fun return await caller(func, *(extras + args), *kw) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/gcsfs/retry.py", line 130, in retry_request return await func(args, kwargs) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/gcsfs/core.py", line 426, in _request async with self.session.request( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/aiohttp/client.py", line 1355, in aenter self._resp: _RetType = await self._coro File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/aiohttp/client.py", line 659, in _request conn = await self._connector.connect( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/aiohttp/connector.py", line 557, in connect proto = await self._create_connection(req, traces, timeout) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/aiohttp/connector.py", line 1002, in _createconnection , proto = await self._create_direct_connection(req, traces, timeout) File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/aiohttp/connector.py", line 1305, in _create_direct_connection transp, proto = await self._wrap_create_connection( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/aiohttp/connector.py", line 1061, in _wrap_create_connection sock = await aiohappyeyeballs.start_connection( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/site-packages/aiohappyeyeballs/impl.py", line 88, in startconnection sock, , _ = await staggered.staggered_race( File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/asyncio/staggered.py", line 144, in staggered_race raise d.exception() File "/share/software/app/anaconda/anaconda3/envs/neuralgcm/lib/python3.10/asyncio/staggered.py", line 116, in run_one_coro assert winner_index is None AssertionError

[weatherforecasterwhai]

Maybe I was wrong. My local server computer is banned from gcs.