"Filter error: unimplemented filter encountered" when installing from conda-forge

[bertcoerver]

Solution to issue cannot be found in the documentation.

• [X] I checked the documentation.

Issue

When I install netcdf4 using conda, I'm unable to open certain datasets, while when using pip to install netcdf4, they do open. According to people over at netcdf4-python (see issue #1182), this is because the binary wheels have support for the optional szip compression filter, while apparently the conda package does not. Although szip has a restrictive license on re-distribution, the binary wheels use libaec which is a drop-in replacement for szip with better licensing.

Creating a new conda environment using:

conda create -n test1_hdf --yes -c conda-forge python netcdf4
conda activate test1_hdf

and then running the following code in Python:

import netCDF4
ds = netCDF4.Dataset(r"/Users/hmcoerver/Local/collect_test/PROBAV/PROBAV/PROBAV_S5_TOC_X21Y04_20210701_100M_V101.HDF5")
ds["LEVEL3/NDVI"]["NDVI"][...]

I get the following error (I can see the metadata of the file, it's just the opening of the actual data with [...] that causes the error):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/netCDF4/_netCDF4.pyx", line 4739, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 5688, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 1965, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Filter error: unimplemented filter encountered

When I create a conda environment like this:

conda create -n test2_hdf --yes -c conda-forge python
conda activate test2_hdf
pip install netcdf4

The code works fine.

Installed packages

# Name                    Version                   Build  Channel
bzip2                     1.0.8                h3422bc3_4    conda-forge
c-ares                    1.18.1               h3422bc3_0    conda-forge
ca-certificates           2022.6.15            h4653dfc_0    conda-forge
cftime                    1.6.1           py310h611a7d1_0    conda-forge
curl                      7.83.1               h7965298_0    conda-forge
hdf4                      4.2.15               hc683e77_4    conda-forge
hdf5                      1.12.2          nompi_h33dac16_100    conda-forge
jpeg                      9e                   he4db4b2_2    conda-forge
krb5                      1.19.3               he492e65_0    conda-forge
libblas                   3.9.0           16_osxarm64_openblas    conda-forge
libcblas                  3.9.0           16_osxarm64_openblas    conda-forge
libcurl                   7.83.1               h7965298_0    conda-forge
libcxx                    14.0.6               h04bba0f_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h642e427_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           11_3_0_hd922786_25    conda-forge
libgfortran5              11.3.0              hdaf2cc0_25    conda-forge
liblapack                 3.9.0           16_osxarm64_openblas    conda-forge
libnetcdf                 4.8.1           nompi_h996a5af_104    conda-forge
libnghttp2                1.47.0               h519802c_1    conda-forge
libopenblas               0.3.21          openmp_hc731615_2    conda-forge
libsqlite                 3.39.2               h2c9beb0_1    conda-forge
libssh2                   1.10.0               h7a5bd25_3    conda-forge
libzip                    1.9.2                h76ab92c_1    conda-forge
libzlib                   1.2.12               ha287fd2_2    conda-forge
llvm-openmp               14.0.4               hd125106_0    conda-forge
ncurses                   6.3                  h07bb92c_1    conda-forge
netcdf4                   1.6.0           nompi_py310h9d42ac0_101    conda-forge
numpy                     1.23.2          py310h127c7cf_0    conda-forge
openssl                   3.0.5                h7aea29f_1    conda-forge
pip                       22.2.2             pyhd8ed1ab_0    conda-forge
python                    3.10.6          hae75cb6_0_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1.2                h46ed386_0    conda-forge
setuptools                65.3.0          py310hbe9552e_0    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
tzdata                    2022c                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge

Environment info

active environment : test1_hdf
    active env location : /Users/hmcoerver/opt/miniconda3/envs/test1_hdf
            shell level : 2
       user config file : /Users/hmcoerver/.condarc
 populated config files : 
          conda version : 4.13.0
    conda-build version : 3.21.9
         python version : 3.9.12.final.0
       virtual packages : __osx=12.5.1=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/hmcoerver/opt/miniconda3  (writable)
      conda av data dir : /Users/hmcoerver/opt/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/hmcoerver/opt/miniconda3/pkgs
                          /Users/hmcoerver/.conda/pkgs
       envs directories : /Users/hmcoerver/opt/miniconda3/envs
                          /Users/hmcoerver/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.13.0 requests/2.28.1 CPython/3.9.12 Darwin/21.6.0 OSX/12.5.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

[ocefpaf]

You are reading an HDF5 file that may have been compressed with szip. https://github.com/conda-forge/hdf5-feedstock/pull/179 should fix that but for now we do not support that filter

[wpreimes]

I have a similar issue (same error message) with the latest 2 versions of netCDF4 from conda-forge (v1.6.2 and v1.6.3) when writing string variables to a netcdf file.

Creating a conda environment via conda create -c conda-forge -n nc python=3.10 netCDF4=1.6.3

and then running the following python script

from netCDF4 import Dataset
import numpy as np

with Dataset('/tmp/test.nc', 'w') as nc:
    nc.createDimension('loc', 3)
    dat = nc.createVariable('var1', str, ('loc',), zlib=True)
    dat[:] = np.array(['one', 'two', 'three'])

raises an error RuntimeError: NetCDF: Filter error: bad id or parameters or duplicate filter.

The same script works fine with netcdf4<=1.6.1 from conda-forge, and it also works fine when I set zlib=False for the latest 2 versions . But the new versions from the defaults channel (i.e. 1.6.2 and 1.6.3) always work fine (even with zlib=True) so I assume this is not intended behavior of the conda-forge version of the package.

For now I use netCDF4 from defaults as a workaround.

[xylar]

@dopplershift, any insight into this? Or suggestions for who to check with? Are we not configuring something correctly here?

[dopplershift]

You can't apply compression to string variables. This never worked, but was made a hard error by recent netcdf-c releases. See this discussion for more info.

Nothing for us to do here, so closing.

[xylar]

Thanks @dopplershift, that's helpful. Odd that the version from defaults isn't showing this error. That actually seems like a problem with the version on defaults.

[dopplershift]

@xylar That's easy: defaults is on 4.8.1, the PR that made the change, Unidata/netcdf-c#2231, wasn't released until netcdf-c (libnetcdf) 4.9.0.

[xylar]

Ah, I see. So nothing to do with the version of netcdf4.

[wpreimes]

Thanks for clarifying