Incompatibility between h5py and netCDF4 (HDF5SAF and netCDF epygram formats)

[AlexandreMary]

Noticed on Ubuntu 24.04 - installing netCDF4 and h5py via pip3 in a virtualenv. Opening a netCDF file fails with error:

OSError: [Errno -101] NetCDF: HDF error

Sébastien, can you have a look ?

[SebastienRietteMTO]

Using virtualenv on Ubuntu 22.04 leads to a similar problem:

python3 -m venv test_nch5 . test_nch5/bin/activate pip3 install netCDF4 pip3 install h5py python3 -c "import netCDF4, h5py; netCDF4.Dataset('.../dims_3D.nc')" #OK python3 -c "import h5py, netCDF4; netCDF4.Dataset('.../dims3D.nc')" #Broken python3 -c "import numpy, h5py, netCDF4; print(numpy.__version__, h5py.__version__, netCDF4.__version_\)" #returns 2.1.1 3.11.0 1.7.1.post2

rm -rf test_nch5

But, specifying the version to use (to mimic the installed versions outside of virtualenv) seems to work:

python3 -m venv test_nch5 . test_nch5/bin/activate pip3 install numpy==1.21.5 netCDF4==1.5.8 h5py==3.6.0 python3 -c "import netCDF4, h5py; netCDF4.Dataset('.../dims_3D.nc')" #OK python3 -c "import h5py, netCDF4; netCDF4.Dataset('.../dims_3D.nc')" #OK

I am waiting for access on a 24.04 machine to perform further tests and identify the versions to use.

[SebastienRietteMTO]

On Ubuntu 24.04, the pre-installed versions are numpy: 1.26.4 h5py: 3.10.0 netCDF4: 1.6.5

In a virtualenv, we can force to install these versions using pip3 install numpy==1.26.4 netCDF4==1.6.5 h5py==3.10.0

version 3.11.0 of h5py seems OK version 2.1.1 of numpy seems to need version 1.7.1 of netCDF4 version 1.7.1 of netCDF4 is incompatible with version 3.10.0 or 3.11.0 of h5py

Building h5py and/or netCDF4 instead of using the wheel also suppresses the problem: pip3 install --no-binary netCDF4 netCDF4 and/or pip3 install --no-binary h5py h5py

Alexandre, can you try one of these two solutions in your environment?