HDF5 database file prone to corruption

sktime / mlaut

Other

24 stars 5 forks source link

HDF5 database file prone to corruption #25

Closed ViktorKaz closed 5 years ago

ViktorKaz commented 6 years ago

H5 files seem to be unreliable and prone to corruption.

The following errors have been observed:

h5 file becomes corrupt when trying to delete multiple groups via HDF View.
h5 file becomes corrupt after making predictions.

ViktorKaz commented 5 years ago

Other databases could be supported in the future.

fkiraly commented 5 years ago

Re-opening as corruption has been observed (discussion on Jan 22, 2019) to occur too when experiments interrupted during writing of results or models (not addressed).

ViktorKaz commented 5 years ago

This seems to be a limitation of the HDF5 format. The file is prone to corruption if not closed properly, e.g. if the orchestrator task is killed. New design allows to implement different backends for loading and saving data.