vol2bird fails to read some german polar volume files

[BerendWijers]

I'm having a few files which are giving me issues. However, the error description is relatively general (I think) and I would love to get some more information from vol2bird on which step it actually fails. Considering the RSL fails - I would in this case like to see at which step and or why RSL is failing.

The error I am receiving is: Unknown input file type. File <DEBOO_pvol_20161128T1230_10132.h5> is not recognized by RSL. critical error, cannot open file DEBOO_pvol_20161128T1230_10132.h5 Error: failed to read radar volume Extra information: os: Ubuntu 18.04.3 LTS vol2bird version: 0.3.20 (24-oct-2018) -94/96 files (DEBOO 2016/11/28) worked without a problem. -2/96 files provide the aforementioned error -I can open these two files with hdfcompass and h5py -I compared erroring pvol with correct pvols within the same day and radar

attributes are identical (except time)

However, I do not have the slightest idea where to start looking inside the PVOL's to figure out what causes RSL to determine that this file is a non RSL supported file. This is why I would like something to point me a bit closer to the actual issue (error) we are facing.

Any suggestions where I can start looking?

Also - I can of course supply these files.

[adokter]

Hi @BerendWijers, what information do you get if you run is.pvolfile() on this file in bioRad?

I would first update vol2bird (0.3.20 is quite old, we are now at 0.5.x), newer versions should not give you an (incorrect) RSL warning anymore. The issue has nothing to do with RSL, as RSL is for reading US NEXRAD data. Somehow something goes wrong when reading the polar volume file with the library that reads european ODIM files (library Rave). It might just be that these two files are corrupted.

Also try to process the file with the latest development version of bioRad, which is fully up to date. To install run:

library(devtools)
install_github("adokter/bioRad")
library(bioRad)
update_docker()

and then try to process your file with calculate_vp(your_file_here).

[BerendWijers]

Hi @adokter

The following commands are all run in succession.

I've updated the bioRad installation to the latest version.

> packageVersion("bioRad")
[1] ‘0.5.0.9289’

I've updated vol2bird to the latest version

> vol2bird_version()
[1] ‘0.5.0’

Then running determining if it is indeed a pvol file.

 > is.pvolfile(error_pvol_paths[[1]])
[1] TRUE

read pvolfile

> error_pvol = read_pvolfile(error_pvol_paths[[1]])
Error: Unable to read dataset.
Not all required filters available.
Missing filters: deflate

Then something strange happens in which I suspect that on error read_pvolfile() does not close the hdf again.

> is.pvolfile(error_pvol_paths[[1]])
[1] TRUE
Warning messages:
1: In h5checktypeOrOpenLoc(file, readonly = TRUE, native = native) :
  An open HDF5 file handle exists. If the file has changed on disk meanwhile, the function may not work properly. Run 'h5closeAll()' to close all open HDF5 object handles.
2: In h5checktypeOrOpenLoc(file, readonly = TRUE, native = native) :
  An open HDF5 file handle exists. If the file has changed on disk meanwhile, the function may not work properly. Run 'h5closeAll()' to close all open HDF5 object handles.
3: In h5checktypeOrOpenLoc(file, readonly = TRUE, native = native) :
  An open HDF5 file handle exists. If the file has changed on disk meanwhile, the function may not work properly. Run 'h5closeAll()' to close all open HDF5 object handles.

Closing files again

> rhdf5::h5closeAll()

Trying to determine if this is indeed a pvolfile, now without error

> is.pvolfile(error_pvol_paths[[1]])
[1] TRUE

Trying another way of determining if this is indeed a pvolfile, in which it says its not

> is.pvol(error_pvol_path[[1]])
[1] FALSE

Now trying to generate a vertical profile:

> calculate_vp(error_pvol_paths[[1]],"../deboo_vp_20161128T1230Z_0x7_10132.h5")
Loaded user configuration file 'options.conf' ...
Warning: failed to read file deboo_pvol_20161128T1230Z_0x7_10132.h5 in ODIM format, ignoring.
 2020/01/22 12:09:08 :    CRITICAL : pvol == NULL (polarvolume.c:903)
Error in calculate_vp(error_pvol_paths[[1]], "../deboo_vp_20161128T1230Z_0x7_10132.h5") : 
  failed to run vol2bird

[adokter]

Hi @BerendWijers, I've moved part of this issue to the bioRad repository, see issue https://github.com/adokter/bioRad/issues/291

The error

Error: Unable to read dataset.
Not all required filters available.
Missing filters: deflate

seems to be related to missing zlib decompression support in the hdf5 library and/or RAVE library used by both vol2bird and bioRad to read ODIM hdf5 files, see https://rdrr.io/bioc/rhdf5/src/R/h5filters.R, or potentially your files are corrupted.

I'm having a few files which are giving me issues.

Could you give an indication how common this problem is?

[BerendWijers]

Hi @adokter

Thank you for the clarification and division of error(s).

Tomorrow I will try to run some tests on these files to determine if they are corrupted or not (as best as I can).

Also - thank you so much for the explanation of the error.

For the frequency of errors: Run stats:

n PVOL total_processed	n PVOL failed
264960	54

[adokter]

That's only 0.02% of your data set, it may be quite likely these files got corrupted somehow during the various transfer processes. Let me know if if it turns out that there is a consistent batch of 54 files of the same radar or from the same time window with this issue, instead of these errors occurring randomly once in a while. Closing for now.

[adokter]

Hi @BerendWijers, could you sent me one file for which you had this problem, it would be helpful to debug the related issue we identified for bioRad (https://github.com/adokter/bioRad/issues/291)

[BerendWijers]

Hi @adokter

Two erroring pvols can be found here

Thank you for investigating this.