Unidata / netcdf4-python

netcdf4-python: python/numpy interface to the netCDF C library
http://unidata.github.io/netcdf4-python
MIT License
754 stars 262 forks source link

test_compress_blosc failure when netcdf is compiled without blosc support. #1171

Closed opoplawski closed 2 years ago

opoplawski commented 2 years ago

Testing updating the Fedora netcdf4-python library to 1.6.0 with netcdf 4.9.0:

+ cd test
+ export NO_NET=1
+ NO_NET=1
++ echo ../build/lib.linux-x86_64-3.11
+ PYTHONPATH=../build/lib.linux-x86_64-3.11
+ /usr/bin/python3 run_all.py
........................F...........................................................................
======================================================================
FAIL: runTest (tst_compression_blosc.CompressionTestCase.runTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/orion/fedora/netcdf4-python/netcdf4-python-1.6.0rel/test/tst_compression_blosc.py", line 53, in runTest
    assert f.variables['data_lz'].filters() == dtest
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

----------------------------------------------------------------------
Ran 100 tests in 163.180s

FAILED (failures=1)
not running tst_dap.py ...

netcdf4-python version: 1.6.0
HDF5 lib version:       1.12.1
netcdf lib version:     4.9.0
numpy version           1.22.0
cython version          0.29.30
compressed lossy with shuffle and standard quantization =  187828  max err =  0.00024413424868552358
compressed lossy with shuffle and alternate quantization =  155437  max err =  0.00048827335259193916
compressed lossy with shuffle and alternate quantization =  163172  max err =  0.00024413952399326977

Now while this notified my that my netcdf build was compiled without blosc support, it seems like this test should not fail but be skipped with a message indicating missing blosc support. Once I compiled netcdf with blosc support the test passed.

jswhit commented 2 years ago

The test is only run if netcdf_filter.h has nc_def_var_blosc defined. That means either the netcdf-c build did actually detect blosc and use it, or the function is defined even if blosc is not available. If it's the latter, that is unfortunate, since I don't know of any other way of detecting whether the filter is available (other than trying to call the function and checking the error code).

jswhit commented 2 years ago

Note that if you set the env var NO_PLUGINS, then none of the external compression filter tests will be run.

opoplawski commented 2 years ago

So, from a quick glance at netcdf_filter.h - nc_def_var_blosc will always be defined. But it looks like there is:

/* See if filter is available */
EXTERNL int nc_inq_filter_avail(int ncid, unsigned id);

Perhaps you could leverage that?

jswhit commented 2 years ago

Right - good idea. Implementing this in https://github.com/Unidata/netcdf4-python/pull/1173

jswhit commented 2 years ago

Closed by PR #1173