search

pydata / xarray

N-D labeled arrays and datasets in Python
https://xarray.dev
Apache License 2.0
3.59k stars 1.08k forks source link

deprecate pynio backend #4491

Closed mathause closed 1 year ago

mathause commented 4 years ago

We are currently not testing with the newest version of netCDF4 because it is incompatible with pynio (the newest version is 1.5.4, we are at 1.5.3). This is unlikely to be fixed, see conda-forge/pynio-feedstock#90.

Therefore we need to think how to setup the tests so we use the newest version of netCDF4. Maybe just remove it from py38.yml?

And long term what to do with the pynio backend? Deprecate? Move to an external repo?

shoyer commented 4 years ago

And long term what to do with the pynio backend? Deprecate? Move to an external repo?

Yes, in the long term I think the pynio backend will need to be moved outside of xarray, along with backends for other unmaintained projects such as pydap. This is one of goals of the backend refactor effort lead by @alexamici and @aurghs.

Until that happens, if new versions of netCDF4 and pynio are incompatible, my preferred work-around would be to reduce test coverage for pynio. It's not being developed actively, so there aren't strong reasons to maintain test coverage with every version of Python.

mathause commented 4 years ago

In py36-min-all-deps.yml we already test the newest version of pynio (1.5.5). My suggestion is therefore:

Keep pynio in

Remove pynio from:

Do you agree?

keewis commented 4 years ago

not sure if we should repurpose py38-all-but-dask for that, right now the only difference between it and py38 is that dask (and all libraries that depend on it) are missing, which is intentional.

Would it be enough to only test using py36-min-all-deps and add a warning to the docs stating that pynio conflicts with netcdf4>1.5.3, and maybe also that the pynio backend will be deprecated soon? Same for pydap, I guess.

dcherian commented 4 years ago

Would it be enough to only test using py36-min-all-deps and add a warning to the docs stating that pynio conflicts with netcdf4>1.5.3, and maybe also that the pynio backend will be deprecated soon? Same for pydap, I guess.

This sounds fine to me.

mathause commented 4 years ago

Would it be enough to only test using py36-min-all-deps and add a warning to the docs stating that pynio conflicts with netcdf4>1.5.3, and maybe also that the pynio backend will be deprecated soon? Same for pydap, I guess.

Yes, fine for me - check #4492

mathause commented 4 years ago

4492 is merged. I changed the title to reflect that pynio should eventually be moved to an external repo but keep the issue open to have the discussion in place. I hope that this should not be too hard after the backend refactor.

mathause commented 3 years ago

We just bumped the the minimum version of python-netCDF4 to 1.5 - we won't be able to bump it to 1.6 because of this. So we'll need to decide what to do with pynio before and properly deprecate it.

Worth noting though, that there is no python-netCDF4 version 1.6 at the moment, so this is not all that urgent. netCDF4 has not released a minor version in a long time (March 2019) (but they have done 12 patch releases).

aurghs commented 3 years ago

If you want I can take care to move pynio to an external repository, but we should decide where.

mathause commented 3 years ago

That would be nice. I would probably call it "xarray-pynio" (?) and move it to the "xarray-contrib" organisation. However, I think this needs a higher-level decision.

keewis commented 3 years ago

I agree, that would probably be the best place to put it. We might want do the same for pydap (which also seems to be unmaintained), too.

dcherian commented 3 years ago

I would probably call it "xarray-pynio" (?) and move it to the "xarray-contrib" organisation.

I think this is OK. The README should say that it is totally unmaintained.

cc @khallock for awareness.

fulminemizzega commented 2 years ago

Hello, sorry to bother you here, can you test again with newer pynio builds? I think pynio-feedstock#90 is now solved.

mathause commented 2 years ago

We can uncomment here:

https://github.com/pydata/xarray/blob/305533d585389f7240ae2383a323337d4761d33a/ci/requirements/environment.yml#L36

which adds it back to the test suite. As the version does not change, the new build should also be picked up from

https://github.com/pydata/xarray/blob/305533d585389f7240ae2383a323337d4761d33a/ci/requirements/py38-min-all-deps.yml#L42

(it's nice to see that the build for pynio is fixed, but the package itself does not seem to be actively maintained).

fulminemizzega commented 2 years ago

Reading here https://www.pyngl.ucar.edu/Nio.shtml it says that PyNIO is in "maintenance mode", what that means is not clear to me. There are no commits since 2019 to the the pynio repo https://github.com/NCAR/pynio.

dcherian commented 2 years ago

6559 is dropping PyNIO because I can't create a min-all-deps environment otherwise. We could try adding it back in one of the other environments.

fulminemizzega commented 2 years ago

6559 is dropping PyNIO because I can't create a min-all-deps environment otherwise. We could try adding it back in one of the other environments.

I've downloaded a copy of min-all-deps.yml from this repo and run micromamba create -f min-all-deps.yml in a fedora container and it looks fine:

(xarray-tests) [root@f5f875d59f07 ~]# micromamba list
List of packages in environment: "/root/micromamba/envs/xarray-tests"

  Name                   Version       Build                   Channel
────────────────────────────────────────────────────────────────────────────
  _libgcc_mutex          0.1           conda_forge             conda-forge
  _openmp_mutex          4.5           2_gnu                   conda-forge
  affine                 2.3.1         pyhd8ed1ab_0            conda-forge
  antlr-python-runtime   4.7.2         py38h578d9bd_1003       conda-forge
  asciitree              0.3.3         py_2                    conda-forge
  attrs                  21.4.0        pyhd8ed1ab_0            conda-forge
  beautifulsoup4         4.11.1        pyha770c72_0            conda-forge
  bokeh                  2.3.3         py38h578d9bd_0          conda-forge
  boost-cpp              1.72.0        h8e57a91_0              conda-forge
  boto3                  1.13.26       pyh9f0ad1d_0            conda-forge
  botocore               1.16.26       pyh9f0ad1d_0            conda-forge
  bottleneck             1.3.4         py38h3ec907f_0          conda-forge
  brotlipy               0.7.0         py38h0a891b7_1004       conda-forge
  bzip2                  1.0.8         h7f98852_4              conda-forge
  c-ares                 1.18.1        h7f98852_0              conda-forge
  ca-certificates        2021.10.8     ha878542_0              conda-forge
  cairo                  1.16.0        hcf35c78_1003           conda-forge
  cartopy                0.17.0        py38h9cf8511_1015       conda-forge
  cdat_info              8.2.1         pyhd8ed1ab_2            conda-forge
  cdms2                  3.1.5         py38h11a58eb_3          conda-forge
  cdtime                 3.1.4         py38hac60b08_2          conda-forge
  certifi                2021.10.8     py38h578d9bd_2          conda-forge
  cf-units               3.0.1         py38h6c62de6_2          conda-forge
  cffi                   1.15.0        py38h3931269_0          conda-forge
  cfgrib                 0.9.8.5       pyhd8ed1ab_0            conda-forge
  cfitsio                3.470         h2e3daa1_7              conda-forge
  cftime                 1.2.1         py38hab2c0dc_1          conda-forge
  charset-normalizer     2.0.12        pyhd8ed1ab_0            conda-forge
  click                  7.1.2         pyh9f0ad1d_0            conda-forge
  click-plugins          1.1.1         py_0                    conda-forge
  cligj                  0.7.2         pyhd8ed1ab_1            conda-forge
  cloudpickle            2.0.0         pyhd8ed1ab_0            conda-forge
  coverage               6.3.2         py38h0a891b7_2          conda-forge
  coveralls              3.3.1         pyhd8ed1ab_0            conda-forge
  cryptography           36.0.2        py38h2b5fc30_1          conda-forge
  curl                   7.76.1        h979ede3_1              conda-forge
  cycler                 0.11.0        pyhd8ed1ab_0            conda-forge
  cytoolz                0.11.2        py38h0a891b7_2          conda-forge
  dask                   2.30.0        py_0                    conda-forge
  dask-core              2.30.0        py_0                    conda-forge
  distarray              2.12.2        pyhd8ed1ab_2            conda-forge
  distributed            2.30.1        py38h578d9bd_0          conda-forge
  docopt                 0.6.2         py_1                    conda-forge
  docutils               0.15.2        py38h578d9bd_3          conda-forge
  eccodes                2.17.0        h59f7be3_1              conda-forge
  esmf                   8.0.0         nompi_hb0fcdcb_6        conda-forge
  esmpy                  8.0.0         nompi_py38hf0e99fa_1    conda-forge
  execnet                1.9.0         pyhd8ed1ab_0            conda-forge
  expat                  2.4.8         h27087fc_0              conda-forge
  fasteners              0.17.3        pyhd8ed1ab_0            conda-forge
  fontconfig             2.14.0        h8e229c2_0              conda-forge
  freetype               2.10.4        h0708190_1              conda-forge
  freexl                 1.0.6         h7f98852_0              conda-forge
  fsspec                 2022.3.0      pyhd8ed1ab_0            conda-forge
  future                 0.18.2        py38h578d9bd_5          conda-forge
  g2clib                 1.6.0         hf3f1b0b_9              conda-forge
  geos                   3.8.1         he1b5a44_0              conda-forge
  geotiff                1.5.1         h05acad5_10             conda-forge
  gettext                0.19.8.1      h73d1719_1008           conda-forge
  giflib                 5.2.1         h36c2ea0_2              conda-forge
  glib                   2.70.2        h780b84a_4              conda-forge
  glib-tools             2.70.2        h780b84a_4              conda-forge
  h5netcdf               0.8.1         py_0                    conda-forge
  h5py                   2.10.0        nompi_py38h513d04c_102  conda-forge
  hdf4                   4.2.15        h10796ff_3              conda-forge
  hdf5                   1.10.5        nompi_h7c3c948_1111     conda-forge
  hdfeos2                2.20          hca1e490_1001           conda-forge
  hdfeos5                5.1.16        h8b6279f_6              conda-forge
  heapdict               1.0.1         py_0                    conda-forge
  hypothesis             6.46.1        pyhd8ed1ab_0            conda-forge
  icu                    64.2          he1b5a44_1              conda-forge
  idna                   3.3           pyhd8ed1ab_0            conda-forge
  importlib-metadata     4.11.3        py38h578d9bd_1          conda-forge
  importlib_metadata     4.11.3        hd8ed1ab_1              conda-forge
  importlib_resources    5.7.1         pyhd8ed1ab_0            conda-forge
  iniconfig              1.1.1         pyh9f0ad1d_0            conda-forge
  iris                   2.4.0         py38_0                  conda-forge
  jasper                 1.900.1       h07fcdf6_1006           conda-forge
  jinja2                 3.1.1         pyhd8ed1ab_0            conda-forge
  jmespath               0.10.0        pyh9f0ad1d_0            conda-forge
  jpeg                   9e            h166bdaf_1              conda-forge
  json-c                 0.13.1        hbfbb72e_1002           conda-forge
  jsonschema             4.4.0         pyhd8ed1ab_0            conda-forge
  jupyter_core           4.9.2         py38h578d9bd_0          conda-forge
  kealib                 1.4.13        hec59c27_0              conda-forge
  kiwisolver             1.4.2         py38h43d8883_1          conda-forge
  krb5                   1.17.2        h926e7f8_0              conda-forge
  lazy-object-proxy      1.7.1         py38h0a891b7_1          conda-forge
  lcms2                  2.12          hddcbb42_0              conda-forge
  ld_impl_linux-64       2.36.1        hea4e1c9_2              conda-forge
  libaec                 1.0.6         h9c3ff4c_0              conda-forge
  libblas                3.8.0         17_openblas             conda-forge
  libcblas               3.8.0         17_openblas             conda-forge
  libcdms                3.1.2         hf60d256_111            conda-forge
  libcf                  1.0.3         py38h88b7cc0_109        conda-forge
  libcurl                7.76.1        hc4aaa36_1              conda-forge
  libdap4                3.20.6        hd7c4107_2              conda-forge
  libdrs                 3.1.2         hc2e2db3_113            conda-forge
  libdrs_f               3.1.2         hae7e664_111            conda-forge
  libedit                3.1.20191231  he28a2e2_2              conda-forge
  libev                  4.33          h516909a_1              conda-forge
  libffi                 3.4.2         h7f98852_5              conda-forge
  libgcc-ng              11.2.0        h1d223b6_16             conda-forge
  libgdal                3.0.4         h3dfc09a_6              conda-forge
  libgfortran-ng         7.5.0         h14aa051_20             conda-forge
  libgfortran4           7.5.0         h14aa051_20             conda-forge
  libglib                2.70.2        h174f98d_4              conda-forge
  libgomp                11.2.0        h1d223b6_16             conda-forge
  libiconv               1.16          h516909a_0              conda-forge
  libkml                 1.3.0         hd79254b_1012           conda-forge
  liblapack              3.8.0         17_openblas             conda-forge
  libllvm10              10.0.1        he513fc3_3              conda-forge
  libnetcdf              4.7.4         nompi_h9f9fd6a_101      conda-forge
  libnghttp2             1.47.0        h727a467_0              conda-forge
  libnsl                 2.0.0         h7f98852_0              conda-forge
  libopenblas            0.3.10        pthreads_hb3c22a3_5     conda-forge
  libpng                 1.6.37        h21135ba_2              conda-forge
  libpq                  12.3          h255efa7_3              conda-forge
  libspatialite          4.3.0a        h2482549_1038           conda-forge
  libssh2                1.10.0        ha56f1ee_2              conda-forge
  libstdcxx-ng           11.2.0        he4da1e4_16             conda-forge
  libtiff                4.2.0         hdc55705_0              conda-forge
  libuuid                2.32.1        h7f98852_1000           conda-forge
  libwebp-base           1.2.2         h7f98852_1              conda-forge
  libxcb                 1.13          h7f98852_1004           conda-forge
  libxml2                2.9.10        hee79883_0              conda-forge
  libxslt                1.1.33        h31b3aaa_0              conda-forge
  libzlib                1.2.11        h166bdaf_1014           conda-forge
  llvmlite               0.34.0        py38h4f45e52_2          conda-forge
  locket                 1.0.0         pyhd8ed1ab_0            conda-forge
  lxml                   4.6.3         py38hf1fe3a4_0          conda-forge
  lz4-c                  1.9.2         he1b5a44_3              conda-forge
  markupsafe             2.1.1         py38h0a891b7_1          conda-forge
  matplotlib-base        3.3.4         py38h0efea84_0          conda-forge
  msgpack-python         1.0.3         py38h43d8883_1          conda-forge
  nbformat               5.3.0         pyhd8ed1ab_0            conda-forge
  nc-time-axis           1.2.0         pyhd8ed1ab_2            conda-forge
  ncurses                6.3           h27087fc_1              conda-forge
  netcdf-fortran         4.5.2         nompi_h45d7149_104      conda-forge
  netcdf4                1.5.3         nompi_py38heb6102f_103  conda-forge
  numba                  0.51.2        py38hc5bc63f_0          conda-forge
  numcodecs              0.9.1         py38h709712a_2          conda-forge
  numpy                  1.18.5        py38h8854b6b_0          conda-forge
  olefile                0.46          pyh9f0ad1d_1            conda-forge
  openblas               0.3.10        pthreads_h43bd3aa_5     conda-forge
  openjpeg               2.3.1         hf7af979_3              conda-forge
  openssl                1.1.1n        h166bdaf_0              conda-forge
  owslib                 0.25.0        pyhd8ed1ab_0            conda-forge
  packaging              20.0          py_0                    conda-forge
  pandas                 1.1.5         py38h51da96c_0          conda-forge
  partd                  1.2.0         pyhd8ed1ab_0            conda-forge
  patsy                  0.5.2         pyhd8ed1ab_0            conda-forge
  pcre                   8.45          h9c3ff4c_0              conda-forge
  pillow                 8.1.2         py38ha0e1e83_0          conda-forge
  pint                   0.16.1        py_0                    conda-forge
  pip                    22.0.4        pyhd8ed1ab_0            conda-forge
  pixman                 0.38.0        h516909a_1003           conda-forge
  pluggy                 1.0.0         py38h578d9bd_3          conda-forge
  poppler                0.67.0        h14e79db_8              conda-forge
  poppler-data           0.4.11        hd8ed1ab_0              conda-forge
  postgresql             12.3          hc2f5b80_3              conda-forge
  proj                   7.0.0         h966b41f_5              conda-forge
  pseudonetcdf           3.1.0         py_1                    conda-forge
  psutil                 5.9.0         py38h0a891b7_1          conda-forge
  pthread-stubs          0.4           h36c2ea0_1001           conda-forge
  py                     1.11.0        pyh6c4a22f_0            conda-forge
  pycparser              2.21          pyhd8ed1ab_0            conda-forge
  pydap                  3.2.2         pyh9f0ad1d_1001         conda-forge
  pyepsg                 0.4.0         py_0                    conda-forge
  pykdtree               1.3.4         py38h6c62de6_2          conda-forge
  pyke                   1.1.1         pyhd8ed1ab_1004         conda-forge
  pynio                  1.5.5         py38h031d99c_12         conda-forge
  pyopenssl              22.0.0        pyhd8ed1ab_0            conda-forge
  pyparsing              3.0.8         pyhd8ed1ab_0            conda-forge
  pyproj                 2.6.1.post1   py38h7521cb9_0          conda-forge
  pyrsistent             0.18.1        py38h0a891b7_1          conda-forge
  pyshp                  2.3.0         pyhd8ed1ab_0            conda-forge
  pysocks                1.7.1         py38h578d9bd_5          conda-forge
  pytest                 7.1.2         py38h578d9bd_0          conda-forge
  pytest-cov             3.0.0         pyhd8ed1ab_0            conda-forge
  pytest-env             0.6.2         py_0                    conda-forge
  pytest-forked          1.4.0         pyhd8ed1ab_0            conda-forge
  pytest-xdist           2.5.0         pyhd8ed1ab_0            conda-forge
  python                 3.8.13        h582c2e5_0_cpython      conda-forge
  python-dateutil        2.8.2         pyhd8ed1ab_0            conda-forge
  python-fastjsonschema  2.15.3        pyhd8ed1ab_0            conda-forge
  python_abi             3.8           2_cp38                  conda-forge
  pytz                   2022.1        pyhd8ed1ab_0            conda-forge
  pyyaml                 6.0           py38h0a891b7_4          conda-forge
  rasterio               1.1.5         py38h033e0f6_1          conda-forge
  readline               8.1           h46c0cb4_0              conda-forge
  requests               2.27.1        pyhd8ed1ab_0            conda-forge
  s3transfer             0.3.7         pyhd8ed1ab_0            conda-forge
  scipy                  1.5.3         py38h828c644_0          conda-forge
  seaborn                0.11.2        hd8ed1ab_0              conda-forge
  seaborn-base           0.11.2        pyhd8ed1ab_0            conda-forge
  setuptools             59.8.0        py38h578d9bd_1          conda-forge
  shapely                1.7.1         py38ha11d057_1          conda-forge
  six                    1.16.0        pyh6c4a22f_0            conda-forge
  snuggs                 1.4.7         py_0                    conda-forge
  sortedcontainers       2.4.0         pyhd8ed1ab_0            conda-forge
  soupsieve              2.3.1         pyhd8ed1ab_0            conda-forge
  sparse                 0.11.2        py_0                    conda-forge
  sqlite                 3.38.3        h4ff8645_0              conda-forge
  statsmodels            0.13.2        py38h6c62de6_0          conda-forge
  tbb                    2020.2        h4bd325d_4              conda-forge
  tblib                  1.7.0         pyhd8ed1ab_0            conda-forge
  tiledb                 1.7.7         h8efa9f0_3              conda-forge
  tk                     8.6.12        h27826a3_0              conda-forge
  toml                   0.10.2        pyhd8ed1ab_0            conda-forge
  tomli                  2.0.1         pyhd8ed1ab_0            conda-forge
  toolz                  0.11.2        pyhd8ed1ab_0            conda-forge
  tornado                6.1           py38h0a891b7_3          conda-forge
  traitlets              5.1.1         pyhd8ed1ab_0            conda-forge
  typing_extensions      3.7.4.3       py_0                    conda-forge
  tzcode                 2022a         h166bdaf_0              conda-forge
  udunits2               2.2.28        hc3e0081_0              conda-forge
  urllib3                1.25.11       py_0                    conda-forge
  webob                  1.8.7         pyhd8ed1ab_0            conda-forge
  wheel                  0.37.1        pyhd8ed1ab_0            conda-forge
  xarray                 2022.3.0      pyhd8ed1ab_0            conda-forge
  xerces-c               3.2.2         h8412b87_1004           conda-forge
  xorg-kbproto           1.0.7         h7f98852_1002           conda-forge
  xorg-libice            1.0.10        h7f98852_0              conda-forge
  xorg-libsm             1.2.3         hd9c2040_1000           conda-forge
  xorg-libx11            1.7.2         h7f98852_0              conda-forge
  xorg-libxau            1.0.9         h7f98852_0              conda-forge
  xorg-libxdmcp          1.1.3         h7f98852_0              conda-forge
  xorg-libxext           1.3.4         h7f98852_1              conda-forge
  xorg-libxrender        0.9.10        h7f98852_1003           conda-forge
  xorg-renderproto       0.11.1        h7f98852_1002           conda-forge
  xorg-xextproto         7.3.0         h7f98852_1002           conda-forge
  xorg-xproto            7.0.31        h7f98852_1007           conda-forge
  xz                     5.2.5         h516909a_1              conda-forge
  yaml                   0.2.5         h7f98852_2              conda-forge
  zarr                   2.5.0         py_0                    conda-forge
  zict                   2.2.0         pyhd8ed1ab_0            conda-forge
  zipp                   3.8.0         pyhd8ed1ab_0            conda-forge
  zlib                   1.2.11        h166bdaf_1014           conda-forge
  zstd                   1.4.8         hdf46e1d_0              conda-forge
dcherian commented 2 years ago

Yes it works now but not after I updated dask and scipy versions in #6559

fulminemizzega commented 2 years ago

My bad, I did not understand correctly what min-all-deps is for. I really have no clue if this can be solved, I only know that to get pynio to work with newer hdf5 required a change in build config for hdf5 (a compatibility macro) and a compiler flag in pynio.

jhamman commented 1 year ago

I suggest we plan to remove the pynio backend in a future release. The project is very likely dead.

https://github.com/NCAR/pynio/issues/53

headtr1ck commented 1 year ago

Does it make sense to move it to its own repo, seeing that this package is no longer maintained?

jhamman commented 1 year ago

I changed the title of this issue to reflect the current situation. PyNIO is unlikely to have another release and its compatibility with the Xarray and the rest of the ecosystem is quickly waning. If someone decides to pick up PyNIO maintenance in the future, they can provide backend entrypoint.