search

NCAS-CMS / cf-python

A CF-compliant Earth Science data analysis library
http://ncas-cms.github.io/cf-python
MIT License
120 stars 19 forks source link

`test_Data_get_filenames` reports errors in HDF5 threads #527

Open sadielbartholomew opened 1 year ago

sadielbartholomew commented 1 year ago

Some CI jobs have produced output indicating errors in HDF5 threads when test_Data_get_filenamesruns: see report in https://github.com/NCAS-CMS/cf-python/pull/520#issuecomment-1350244307 (with further detail in https://github.com/NCAS-CMS/cf-python/pull/520#issuecomment-1351305513). As deduced by @davidhassell in https://github.com/NCAS-CMS/cf-python/pull/520#issuecomment-1350649482, this is probably not an issue with the cf-python code, rather HDF5 and netCDF-C, but I'm opening this Issue because I/we don't have time to look into it now and we should confirm it is indeed 'not our problem' (and find a way to prevent the errors showing up regardless).

davidhassell commented 1 year ago

I wonder if #531 has fixed this?

sadielbartholomew commented 1 year ago

Thanks for the heads up David, I'll look into it once I've tackled higher-priority work...

sadielbartholomew commented 1 year ago

Still seeing this now with the current main branch. Specifically, running test_Field.py shows (highly repeatedly) the following across multiple test methods, including the initial one, test_Field__add__:

HDF5-DIAG: Error detected in HDF5 (1.14.0) thread 1:
  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute
    major: Attribute
    minor: Can't open object
  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'
    major: Attribute
    minor: Can't open object
  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'
    major: Attribute
    minor: Can't open object
  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #005: H5VLnative_attr.c line 161 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #006: H5Aint.c line 658 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #007: H5Oattribute.c line 502 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
    major: Attribute
    minor: Object not found
davidhassell commented 1 year ago

Which version of netcdf4-python are you using?

sadielbartholomew commented 1 year ago

I am using 1.6.3, specifically with:

$ conda list | grep netcdf
libnetcdf                 4.9.2           nompi_hf3f8848_103    conda-forge
netcdf-flattener          1.2.0              pyh9f0ad1d_0    conda-forge
netcdf-fortran            4.6.0           nompi_h3142581_105    conda-forge
netcdf4                   1.6.3           nompi_py38hd31e641_102    conda-forge

which appears to be the latest version having checked over on https://github.com/Unidata/netcdf4-python. What version are you using and are you seeing this kind of message crop up?