[Bug]: `RuntimeError: NetCDF: Not a valid ID` appears randomly when running operations on the same datasets repeatedly

[tomvothecoder]

What happened?

For some reason, the RuntimeError: NetCDF: Not a valid ID is being thrown at random times when running operations on the same dataset.

This has happened when I call the 1) spatial averaging and 2) temporal averaging APIs.

1) Spatial Averaging API

Example:

ds = xc.open_dataset("some_dataset")

for _ in range(0, 5):
    ds_res = ds.spatial.average(axis=["X", "Y"], weighted=True)
    ds_res_comp = ds_res.compute()

2) Temporal Averaging API

For this API, the RuntimeError stacktrace points to the logic in TemporalAccessor._get_weights(). Temporal averaging should work fine across multiple copies of the same dataset object.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/home/vo13/xCDAT/xcdat_test/validation/v0.6.0/xcdat-cdat-perf-metrics/Untitled-1.py in line 2
      58 #%%
----> 59 ds2 = ds.copy().temporal.departures("tas", freq='month', weighted=True)

File ~/mambaforge/envs/xcdat_test_stable/lib/python3.10/site-packages/xcdat/temporal.py:687, in TemporalAccessor.departures(self, data_var, freq, weighted, keep_weights, reference_period, season_config)
    685 inferred_freq = _infer_freq(ds[self.dim])
    686 if inferred_freq != freq:
--> 687     ds_obs = ds_obs.temporal.group_average(
    688         data_var,
    689         freq,
    690         weighted,
    691         keep_weights,
    692         season_config,
    693     )
    695 ds_climo = ds.temporal.climatology(
    696     data_var,
    697     freq,
   (...)
    701     season_config,
    702 )
    704 # 4. Group the averaged data variable values by the time `freq`.
    705 # ----------------------------------------------------------------------
    706 # This step allows us to perform xarray's grouped arithmetic to
    707 # calculate departures.

File ~/mambaforge/envs/xcdat_test_stable/lib/python3.10/site-packages/xcdat/temporal.py:353, in TemporalAccessor.group_average(self, data_var, freq, weighted, keep_weights, season_config)
    225 """Returns a Dataset with average of a data variable by time group.
    226 
    227 Time bounds are used for generating weights to calculate weighted group
   (...)
    349 }
    350 """
    351 self._set_data_var_attrs(data_var)
--> 353 return self._averager(
    354     data_var,
    355     "group_average",
    356     freq,
    357     weighted=weighted,
    358     keep_weights=keep_weights,
    359     season_config=season_config,
    360 )

File ~/mambaforge/envs/xcdat_test_stable/lib/python3.10/site-packages/xcdat/temporal.py:767, in TemporalAccessor._averager(self, data_var, mode, freq, weighted, keep_weights, reference_period, season_config)
    765     dv = self._average(dv, time_bounds)
    766 elif self._mode in ["group_average", "climatology", "departures"]:
--> 767     dv = self._group_average(dv, time_bounds)
    769 # The original time dimension is dropped from the dataset because
    770 # it becomes obsolete after the data variable is averaged. When the
    771 # averaged data variable is added to the dataset, the new time dimension
    772 # and its associated coordinates are also added.
    773 ds = ds.drop_dims(self.dim)  # type: ignore

File ~/mambaforge/envs/xcdat_test_stable/lib/python3.10/site-packages/xcdat/temporal.py:1132, in TemporalAccessor._group_average(self, data_var, time_bounds)
   1129 self._labeled_time = self._label_time_coords(dv[self.dim])
   1131 if self._weighted:
-> 1132     self._weights = self._get_weights(time_bounds)
   1133     # Weight the data variable.
   1134     dv *= self._weights

File ~/mambaforge/envs/xcdat_test_stable/lib/python3.10/site-packages/xcdat/temporal.py:1216, in TemporalAccessor._get_weights(self, time_bounds)
   1209 # Must be cast dtype from "timedelta64[ns]" to "float64", specifically
   1210 # when using Dask arrays. Otherwise, the numpy warning below is thrown:
   1211 # `DeprecationWarning: The `dtype` and `signature` arguments to ufuncs
   1212 # only select the general DType and not details such as the byte order
   1213 # or time unit (with rare exceptions see release notes). To avoid this
   1214 # warning please use the scalar types `np.float64`, or string notation.`
   1215 if isinstance(time_lengths.data, Array):
-> 1216     time_lengths.load()
   1218 time_lengths = time_lengths.astype(np.float64)
   1220 grouped_time_lengths = self._group_data(time_lengths)

File ~/mambaforge/envs/xcdat_test_stable/lib/python3.10/site-packages/xarray/core/dataarray.py:1122, in DataArray.load(self, **kwargs)
   1104 def load(self, **kwargs) -> Self:
   1105     """Manually trigger loading of this array's data from disk or a
   1106     remote source into memory and return this array.
   1107 
   (...)
   1120     dask.compute
   1121     """
-> 1122     ds = self._to_temp_dataset().load(**kwargs)
   1123     new = self._from_temp_dataset(ds)
   1124     self._variable = new._variable

File ~/mambaforge/envs/xcdat_test_stable/lib/python3.10/site-packages/xarray/core/dataset.py:847, in Dataset.load(self, **kwargs)
    844 chunkmanager = get_chunked_array_type(*lazy_data.values())
    846 # evaluate all the chunked arrays simultaneously
--> 847 evaluated_data = chunkmanager.compute(*lazy_data.values(), **kwargs)
    849 for k, data in zip(lazy_data, evaluated_data):
    850     self.variables[k].data = data

File ~/mambaforge/envs/xcdat_test_stable/lib/python3.10/site-packages/xarray/core/daskmanager.py:70, in DaskManager.compute(self, *data, **kwargs)
     67 def compute(self, *data: DaskArray, **kwargs) -> tuple[np.ndarray, ...]:
     68     from dask.array import compute
...
File src/netCDF4/_netCDF4.pyx:4496, in netCDF4._netCDF4.Variable._getdims()

File src/netCDF4/_netCDF4.pyx:2028, in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: Not a valid ID

Possible answers:

• In _get_weights(), make a copy of time_lengths variable and load that into memory instead of the original time_lengths variable (which might be closed after the first API call, resulting in the Runtime: NetCDF error.

import xarray
import xcdat as xc

dir_path = "/some/dir/path/"
ds = xc.open_mfdataset(dir_path, chunks={"time": "auto"}, add_bounds=['X', 'Y', 'T'], parallel=True)

# `RuntimeError: NetCDF: Not a valid ID`
ds2 = ds.copy().temporal.departures("tas", freq='month', weighted=True)

There is a line of code that calculates time_lengths using time bounds. The time_lengths are then loaded into memory.

https://github.com/xCDAT/xcdat/blob/c83f46e2f35095d00ea5e05dde1142b7a318c719/xcdat/temporal.py#L1209-L1216

What did you expect to happen? Are there are possible answers you came

APIs should run work consistently across multiple copies of the same dataset.

@jasonb5 suggests that it could be the LLNL climate filesystem not behaving well with Xarray dataset I/O.

• [ ] Try the same code out on local machine

Minimal Complete Verifiable Example (MVCE)

Provided above.

Relevant log output

Provided above.

Anything else we need to know?

Related GitHub issues:

• https://github.com/pydata/xarray/issues/1001
• https://github.com/pydata/xarray/issues/5276

Environment

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
alsa-lib                  1.2.8                h166bdaf_0    conda-forge
anyio                     3.7.1              pyhd8ed1ab_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py310h2372a71_4    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
aws-c-auth                0.7.0                hf8751d9_2    conda-forge
aws-c-cal                 0.6.0                h93469e0_0    conda-forge
aws-c-common              0.8.23               hd590300_0    conda-forge
aws-c-compression         0.2.17               h862ab75_1    conda-forge
aws-c-event-stream        0.3.1                h9599702_1    conda-forge
aws-c-http                0.7.11               hbe98c3e_0    conda-forge
aws-c-io                  0.13.28              h3870b5a_0    conda-forge
aws-c-mqtt                0.8.14               h2e270ba_2    conda-forge
aws-c-s3                  0.3.13               heb0bb06_2    conda-forge
aws-c-sdkutils            0.1.11               h862ab75_1    conda-forge
aws-checksums             0.1.16               h862ab75_1    conda-forge
aws-crt-cpp               0.20.3               he9c0e7f_4    conda-forge
aws-sdk-cpp               1.10.57             hbc2ea52_17    conda-forge
babel                     2.13.1             pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
bokeh                     3.3.0              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_9    conda-forge
brotli-bin                1.0.9                h166bdaf_9    conda-forge
brotli-python             1.0.9           py310hd8f1fbe_9    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.20.1               hd590300_1    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cairo                     1.16.0            ha61ee94_1014    conda-forge
cartopy                   0.20.2          py310hb408dcc_6    conda-forge
cdat_info                 8.2.1              pyhd8ed1ab_2    conda-forge
cdms2                     3.1.5           py310h5f4584e_15    conda-forge
cdtime                    3.1.4           py310h1a17f1e_7    conda-forge
cdutil                    8.2.1              pyhd8ed1ab_4    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cf_xarray                 0.8.4              pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py310h2fee648_0    conda-forge
cftime                    1.6.2           py310h1f7b6fc_2    conda-forge
charset-normalizer        3.3.1              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
cloudpickle               3.0.0              pyhd8ed1ab_0    conda-forge
contourpy                 1.1.1           py310hd41b1e2_1    conda-forge
curl                      8.1.2                h409715c_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
cytoolz                   0.12.2          py310h2372a71_1    conda-forge
dask                      2023.10.0          pyhd8ed1ab_0    conda-forge
dask-core                 2023.10.0          pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.8.0           py310hc6cd4ac_1    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distarray                 2.12.2             pyh050c7b8_4    conda-forge
distributed               2023.10.0          pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
esmf                      8.4.0           mpi_mpich_hc592774_104    conda-forge
esmpy                     8.4.0           mpi_mpich_py310h515c5ea_102    conda-forge
exceptiongroup            1.1.3              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
fftw                      3.3.10          nompi_hc118613_108    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.43.1          py310h2372a71_0    conda-forge
freeglut                  3.2.2                h9c3ff4c_1    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
fsspec                    2023.10.0          pyhca7485f_0    conda-forge
future                    0.18.3             pyhd8ed1ab_0    conda-forge
g2clib                    1.6.3                hbecde78_1    conda-forge
genutil                   8.2.1           py310h2eba5c9_4    conda-forge
geos                      3.10.3               h27087fc_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glib                      2.78.0               hfc55251_0    conda-forge
glib-tools                2.78.0               hfc55251_0    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
gst-plugins-base          1.22.0               h4243ec0_2    conda-forge
gstreamer                 1.22.0               h25f0c4b_2    conda-forge
gstreamer-orc             0.4.34               hd590300_0    conda-forge
harfbuzz                  6.0.0                h8e241bc_0    conda-forge
hdf4                      4.2.15               h9772cbc_5    conda-forge
hdf5                      1.12.2          mpi_mpich_h5d83325_1    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
importlib_resources       6.1.0              pyhd8ed1ab_0    conda-forge
ipykernel                 6.13.0          py310hfdc917e_0    conda-forge
ipython                   8.16.1             pyh0d859eb_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jack                      1.9.22               h11f4161_0    conda-forge
jasper                    2.0.33               h0ff4b12_1    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
json5                     0.9.14             pyhd8ed1ab_0    conda-forge
jsonschema                4.19.1             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.7.1           pyhd8ed1ab_0    conda-forge
jupyter_client            8.5.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.4.0           py310hff52083_0    conda-forge
jupyter_server            1.24.0             pyhd8ed1ab_0    conda-forge
jupyterlab                3.4.2              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.25.0             pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.5           py310hd41b1e2_1    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lazy-object-proxy         1.9.0           py310h2372a71_1    conda-forge
lcms2                     2.14                 h6ed2654_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20230125.3      cxx17_h59595ed_0    conda-forge
libaec                    1.1.2                h59595ed_1    conda-forge
libarrow                  12.0.1           h657c46f_7_cpu    conda-forge
libblas                   3.9.0           19_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcap                    2.67                 he9d0100_0    conda-forge
libcblas                  3.9.0           19_linux64_openblas    conda-forge
libcdms                   3.1.2              h2973bc7_118    conda-forge
libcf                     1.0.3           py310hbc577d2_115    conda-forge
libclang                  15.0.7          default_h7634d5b_3    conda-forge
libclang13                15.0.7          default_h9986a30_3    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcups                   2.3.3                h36d4200_3    conda-forge
libcurl                   8.1.2                h409715c_0    conda-forge
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.14                 h166bdaf_0    conda-forge
libdrs                    3.1.2              hf593df3_118    conda-forge
libdrs_f                  3.1.2              h7e76ec7_114    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h28343ad_4    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgcrypt                 1.10.1               h166bdaf_0    conda-forge
libgfortran-ng            13.2.0               h69a702a_2    conda-forge
libgfortran5              13.2.0               ha4646dd_2    conda-forge
libglib                   2.78.0               hebfc3b9_0    conda-forge
libglu                    9.0.0             he1b5a44_1001    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
libgoogle-cloud           2.12.0               h840a212_1    conda-forge
libgpg-error              1.47                 h71f35ed_0    conda-forge
libgrpc                   1.56.2               h3905398_1    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libidn2                   2.3.4                h166bdaf_0    conda-forge
liblapack                 3.9.0           19_linux64_openblas    conda-forge
libllvm14                 14.0.6               hcd5def8_4    conda-forge
libllvm15                 15.0.7               hadd5161_1    conda-forge
libnetcdf                 4.8.1           mpi_mpich_hcd871d9_6    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.3                 hbcd7760_1    conda-forge
libprotobuf               4.23.3               hd1fb520_1    conda-forge
libsndfile                1.2.2                hc60ed4a_1    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsqlite                 3.43.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libsystemd0               253                  h8c4010b_1    conda-forge
libthrift                 0.18.1               h5e4af38_0    conda-forge
libtiff                   4.4.0                h82bc61c_5    conda-forge
libtool                   2.4.7                h27087fc_0    conda-forge
libudev1                  253                  h0b41bf4_1    conda-forge
libudunits2               2.2.28               h40f5838_3    conda-forge
libunistring              0.9.10               h7f98852_0    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.5.0                h79f4944_1    conda-forge
libxml2                   2.10.3               hca2bb57_4    conda-forge
libxslt                   1.1.37               h873f0b0_0    conda-forge
libzip                    1.10.1               h2629f0a_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
llvmlite                  0.40.1          py310h1b8f574_0    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lxml                      4.9.2           py310hbdc0903_0    conda-forge
lz4                       4.3.2           py310h350c4a5_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markupsafe                2.1.3           py310h2372a71_1    conda-forge
matplotlib                3.5.2           py310hff52083_1    conda-forge
matplotlib-base           3.5.2           py310h5701ce4_1    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mistune                   3.0.1              pyhd8ed1ab_0    conda-forge
mpg123                    1.32.3               h59595ed_0    conda-forge
mpi                       1.0                       mpich    conda-forge
mpi4py                    3.1.4           py310h37cc914_0    conda-forge
mpich                     4.0.3              h846660c_100    conda-forge
msgpack-python            1.0.6           py310hd41b1e2_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.33               hf1915f5_5    conda-forge
mysql-libs                8.0.33               hca2cd23_5    conda-forge
nbclassic                 0.5.6              pyhb4ecaf3_1    conda-forge
nbclient                  0.8.0              pyhd8ed1ab_0    conda-forge
nbconvert                 7.9.2              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.9.2              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.9.2              pyhd8ed1ab_0    conda-forge
nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nest-asyncio              1.5.8              pyhd8ed1ab_0    conda-forge
netcdf-fortran            4.6.0           mpi_mpich_h1e13492_2    conda-forge
netcdf4                   1.6.2           nompi_py310h55e1e36_100    conda-forge
notebook-shim             0.2.3              pyhd8ed1ab_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.94                 h1d7d5a4_0    conda-forge
numba                     0.57.1          py310h0f6aa51_0    conda-forge
numpy                     1.24.4          py310ha4c1d20_0    conda-forge
openblas                  0.3.24          pthreads_h7a3da1a_0    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   3.1.4                hd590300_0    conda-forge
orc                       1.9.0                h385abfd_1    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.1.1           py310hcc13569_1    conda-forge
pandoc                    3.1.3                h32600fe_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parallelio                2.5.10          mpi_mpich_h862c5c2_100    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
partd                     1.4.1              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.2.0           py310h454ad03_3    conda-forge
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pixman                    0.42.2               h59595ed_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              3.11.0             pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
proj                      9.0.1                h93bde94_1    conda-forge
prometheus_client         0.17.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
psutil                    5.9.5           py310h2372a71_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio                16.1                 hcb278e6_3    conda-forge
pulseaudio-client         16.1                 h5195f5e_3    conda-forge
pulseaudio-daemon         16.1                 ha8d29e2_3    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyarrow                   12.0.1          py310h0576679_7_cpu    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyproj                    3.4.0           py310hf94497c_0    conda-forge
pyqt                      5.15.9          py310h04931ad_5    conda-forge
pyqt5-sip                 12.12.2         py310hc6cd4ac_5    conda-forge
pyshp                     2.3.1              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         hd12c33a_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.18.1             pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1           py310h2372a71_1    conda-forge
pyzmq                     25.1.1          py310h795f18f_2    conda-forge
qt-main                   5.15.8               h5d23da1_6    conda-forge
rdma-core                 28.9                 h59595ed_1    conda-forge
re2                       2023.03.02           h8c504da_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.30.2             pyhd8ed1ab_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
rpds-py                   0.10.6          py310hcb5633a_0    conda-forge
s2n                       1.3.46               h06160fa_0    conda-forge
scipy                     1.11.3          py310hb13e2d6_1    conda-forge
send2trash                1.8.2              pyh41d4057_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shapely                   1.8.2           py310h7b2ee30_2    conda-forge
sip                       6.7.12          py310hc6cd4ac_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
sparse                    0.14.0             pyhd8ed1ab_0    conda-forge
sqlite                    3.43.2               h2c6b66d_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tblib                     2.0.0              pyhd8ed1ab_0    conda-forge
terminado                 0.17.1             pyh41d4057_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.13               h2797004_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.3.3           py310h2372a71_1    conda-forge
traitlets                 5.12.0             pyhd8ed1ab_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucx                       1.14.1               h64cca9d_5    conda-forge
unicodedata2              15.1.0          py310h2372a71_0    conda-forge
urllib3                   2.0.7              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.8              pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
websocket-client          1.6.4              pyhd8ed1ab_0    conda-forge
wget                      1.20.3               ha35d2d1_1    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xarray                    2023.10.1          pyhd8ed1ab_0    conda-forge
xcb-util                  0.4.0                h516909a_0    conda-forge
xcb-util-image            0.4.0                h166bdaf_0    conda-forge
xcb-util-keysyms          0.4.0                h516909a_0    conda-forge
xcb-util-renderutil       0.3.9                h166bdaf_0    conda-forge
xcb-util-wm               0.4.1                h516909a_0    conda-forge
xcdat                     0.6.0              pyhd8ed1ab_0    conda-forge
xesmf                     0.8.2              pyhd8ed1ab_0    conda-forge
xgcm                      0.8.1              pyhd8ed1ab_0    conda-forge
xkeyboard-config          2.38                 h0b41bf4_0    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.4                h0b41bf4_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xyzservices               2023.10.1          pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zeromq                    4.3.5                h59595ed_0    conda-forge
zict                      3.0.0              pyhd8ed1ab_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

[tomvothecoder]

I don't think this is necessarily an xCDAT issue based on the random occurrence and the related Xarray issues linked in the issue's description.

As Jason mentioned, it might have to do with the I/O on the Climate Program filesystem closing files that are open in Xarray.

[tomvothecoder]

Related to:

• https://github.com/E3SM-Project/e3sm_to_cmip/issues/246
• https://forum.access-hive.org.au/t/netcdf-not-a-valid-id-errors/389/14

To summarise in this thread, it looks like a work-around in netcdf4-python to deal with netcdf-c not being thread safe was removed in 1.6.1. The solution (for now) is to [make sure your cluster only uses 1 thread per worker](https://forum.access-hive.org.au/t/netcdf-not-a-valid-id-errors/389/14).

It seems like some filesystems do not like parallel access to files. The workaround seems to be to set parallel=False (source)