rortizmerino / dask-playground

Other
0 stars 0 forks source link

Failed dask tests using netCDF #2

Open rortizmerino opened 2 months ago

rortizmerino commented 2 months ago

Seems a different netCDF library is required but not sure how to install it. Error message has a version requirements which should already be met. Also seems the library is needed for some compression (zstd, blosc_*)

============================================== short test summary info ==============================================
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4Data::test_compression_encoding[zstd] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='zstd' requires netCDF lib >= 4.9.0 (using 4.9.3-de...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4Data::test_compression_encoding[blosc_lz] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4Data::test_compression_encoding[blosc_lz4] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4Data::test_compression_encoding[blosc_lz4hc] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4Data::test_compression_encoding[blosc_zlib] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4Data::test_compression_encoding[blosc_zstd] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_compression_encoding[zstd] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='zstd' requires netCDF lib >= 4.9.0 (using 4.9.3-de...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_compression_encoding[blosc_lz] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_compression_encoding[blosc_lz4] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_compression_encoding[blosc_lz4hc] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_compression_encoding[blosc_zlib] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
FAILED .venv/lib/python3.10/site-packages/xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_compression_encoding[blosc_zstd] - netCDF4._netCDF4.NetCDF4MissingFeatureException: compression='blosc_*' requires netCDF lib >= 4.9.0 (using 4.9.3...
==== 282 failed, 14811 passed, 2088 skipped, 65 xfailed, 9 xpassed, 1732 warnings, 1 error in 285.62s (0:04:45) =====

real    4m47.798s
user    6m43.869s
sys     0m12.977s
rortizmerino commented 2 months ago

Current latest version available via pip is 'netCDF4==1.7.1.post1'. To get 'netCDF lib >= 4.9.0', there are development install instructions but seem quite cumbersome

  • Clone the github repository. Make sure you either clone recursively, or run git submodule update --init to ensure all the submodules are also checked out.

  • Make sure the dependencies are satisfied (Python 3.8 or later, numpy, Cython, cftime, setuptools, the HDF5 C library, and the netCDF C library). For MPI parallel IO support, an MPI-enabled versions of the netcdf library is required, as is mpi4py. Parallel IO further depends on the existence of MPI-enabled HDF5 or the PnetCDF library.

  • By default, the utility nc-config (installed with netcdf-c) will be run used to determine where all the dependencies live.

  • If nc-config is not in your default PATH, you can set the NETCDF4_DIR environment variable and setup.py will look in $NETCDF4_DIR/bin. You can also use the file setup.cfg to set the path to nc-config, or enter the paths to the libraries and include files manually. Just edit the setup.cfg file in a text editor and follow the instructions in the comments. To disable the use of nc-config, set the env var USE_NCCONFIG to 0. To disable the use of setup.cfg, set USE_SETUPCFG to 0. As a last resort, the library and include paths can be set via environment variables. If you go this route, set USE_NCCONFIG and USE_SETUPCFG to 0, and specify NETCDF4_LIBDIR, NETCDF4_INCDIR, HDF5_LIBDIR and HDF5_INCDIR. If the dependencies are not found in any of the paths specified by environment variables, then standard locations (such as /usr and /usr/local) are searched.

  • if the env var NETCDF_PLUGIN_DIR is set to point to the location of the netcdf-c compression plugins built by netcdf >= 4.9.0, they will be installed inside the package. In this case HDF5_PLUGIN_PATH will be set to the package installation path on import, so the extra compression algorithms available in netcdf-c >= 4.9.0 will automatically be available. Otherwise, the user will have to set HDF5_PLUGIN_PATH explicitly to have access to the extra compression plugins.

  • run pip install -v . (as root if necessary)

  • run the tests in the 'test' directory by running python run_all.py.

Source: https://unidata.github.io/netcdf4-python/#netCDF4