[Hdf5TreeModel] Concurrent access to hdf5 files issues.

[dnaudet]

I encountered a problem I managed to work around a while ago, but using the Hdf5TreeModel reproduces it. (disclaimer : I probably missed something important in h5py/python doc though, so...)

tl;dr :

https://github.com/h5py/h5py/blob/master/examples/multiprocessing_example.py

Remember that when a process is fork()ed, the child inherits the HDF5 state from its parent, which can be dangerous if you already have a file open. Trying to interact with the same file on disk from multiple processes results in undefined behavior.

Maybe we should consider closing the files once we're done populating the model, and opening them only when needed. Also maybe provide a way for the user to programmaticaly lock the model. Or warn the users that they should disable the view and not do anything on the model if the file is being updated elsewhere.

Long version :

I have an hdf5 file ("master") that contains only external links to other files ("subfiles"). I use the multiprocessing python lib to open the files, read from the files (opened in their respective subprocess) and process the data. I have tried the following scenarii :

• if i dont open the files at all in the main python process before starting the processes : no problem.
• if i open one of the subfiles directly (not following the link from the master file) in the main python thread (without accessing a dataset) and run the processes without closing it first : i get random errors (of two different kinds!) when a process tries to access the file.
• if i access one of the subfiles via the master file in the main python thread, get a reference to one of its datasets, close the file and run the processes (so the reference still exists in the main process even tho it cannot be used anymore) : i get the errors.
• same case a above, but this time i delete the reference before starting the threads : no problem.
• if i access one of the subfiles directly in the main python thread, get a reference to one of its datasets, close the file and run the processes (WITHOUT deleting the reference!) : no problem.

It seems that h5py (or libhdf5) opens external links with mode 'r+' (even though the master is opened as read only), but i've tried both modes when reading the subfiles directly and it didn't change anything.

If access to files is synchronized (a lock for each file) the problems seem to disappear (in all described cases). But, of course if i have a Hdf5TreeModel loaded with the master file at the same time, i get those access errors again, because i cant keep the model from accessing the file.

[vasole]

I think we should start using HDF5 1.10.x as soon as possible in order to see how the SWMR can help on this.