ARPA-SIMC / arkimet

A set of tools to organize, archive and distribute data files.
Other
15 stars 5 forks source link

Failure to import odim file when there is an empty file in dataset #315

Open dcesari opened 1 year ago

dcesari commented 1 year ago

After a system crash, an odim segment folder ended containing a -probably unindexed- empty file at the end of the sequence, such as in this simulated example:

[dcesari@Cesari-L-SMR scan_odim]$ ls -l radar_vol/2023/08-23.odimh5
totale 244
-rw-r--r-- 1 dcesari dcesari 247817 25 ago 11.49 000000.odimh5
-rw-r--r-- 1 dcesari dcesari      0 25 ago 12.11 000001.odimh5

This situation prevented further import of data for the same segment since all the data were dispatched to error with the message cannot open file /path/to/radar_vol/2023/08-23.odimh5/000001.odimh5: File exists.

Could we handle automatically such situations in a safe way?

Actually I cannot even fix the situation with an arki-check -f, it recognizes the spurious file, but it crashes when trying to open it.

The situation can be reproduced by touch-ing the following file at the end of an odim sequence, but I can provide a test case if needed.