search

githubgphl / imginfo

Extract useful meta-data for MX data processing
Other
0 stars 5 forks source link

Catch error codes from HDF5 libraries #6

Open graeme-winter opened 4 months ago

graeme-winter commented 4 months ago

e.g. 

ws448 ap :) $ imginfo -v -v /dls/mx-scratch/yqf46943/error-h5/20240429_BMS-0003_11_1_1_master.h5 > /tmp/out
HDF5-DIAG: Error detected in HDF5 (1.10.7) thread 1:
  #000: H5Dio.c line 198 in H5Dread(): can't read data
    major: Dataset
    minor: Read failed
  #001: H5Dio.c line 463 in H5D__read(): unable to set up type info
    major: Dataset
    minor: Unable to initialize object
  #002: H5Dio.c line 977 in H5D__typeinfo_init(): unable to convert between src and dest datatype
    major: Dataset
    minor: Feature is unsupported
  #003: H5T.c line 4680 in H5T_path_find(): can't find datatype conversion path
    major: Datatype
    minor: Can't get value
  #004: H5T.c line 4893 in H5T__path_find_real(): no appropriate function for conversion path
    major: Datatype
    minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.10.7) thread 1:
  #000: H5Dio.c line 198 in H5Dread(): can't read data
    major: Dataset
    minor: Read failed
  #001: H5Dio.c line 463 in H5D__read(): unable to set up type info
    major: Dataset
    minor: Unable to initialize object
  #002: H5Dio.c line 977 in H5D__typeinfo_init(): unable to convert between src and dest datatype
    major: Dataset
    minor: Feature is unsupported
  #003: H5T.c line 4680 in H5T_path_find(): can't find datatype conversion path
    major: Datatype
    minor: Can't get value
  #004: H5T.c line 4893 in H5T__path_find_real(): no appropriate function for conversion path
    major: Datatype
    minor: Unable to initialize object

however imginfo ran to completion though printed a warning that something was missing: feels like this should have been caught by imginfo

We are happy to look at this also

CV-GPhL commented 4 months ago

Depending on the environment where it is run (in a test environment to check certain compliances versus within automatically triggered pipelines) a different behaviour might be considers: what constitutes a fatal error on one might not be one in the other.

Within a processing pipeline a user has options to override such issues. This is a crucial mechanism we use in autoPROC: there can be breaking changes in HDF5 datasets inbetween software releases that would stop all processing jobs if they become a fatal error, but either autoPROC recovers automatically (using other defaults etc) or allows users to provide a temporary fixup directly on the command-line.

Anyway, adding an option (--stop-on-error or such) that would trigger your suggested behaviour seems like a good idea, yes.