search

G-Node / nix

Neuroscience information exchange format
https://readthedocs.org/projects/nixio/
Other
66 stars 36 forks source link

Lock bit when opening file for writing #625

Open achilleas-k opened 8 years ago

achilleas-k commented 8 years ago

It might be a nice idea to have a lock attribute set on the file when it is open for writing, to protect against multiple instances writing to a file simultaneously.

This would protect against accidentally running two processes that write to the same file, which can cause corruption (this just happened), or even against two different people writing to the same file on a remote machine (on a common network share, for instance).

What do you think?

jgrewe commented 8 years ago

did this happen with the hdf5 backend?! This should be capable of parallel access from multiple processes.

achilleas-k commented 8 years ago

This happened with the old (pre pycore) version of nixpy. I'm not sure whether HDF5 can handle concurrent writes to the same group/dataset. If you say it should, maybe something else caused the corruption.

ajkswamy commented 8 years ago

Hi Jan

We were using a nix file that was on machine A which was also mounted on machine B using sshfs over the local network. A process on machine A opened and started writing into the nix file. Before it finished writing and closed the nix file, a process on machine B opened and started writing into the mounted nix file. Both processes finished without error. However, in the resulting nix file, a section that was written into by both the processes could no longer be opened (RuntimeError:...checksum failed...). The same error was seen both when using nix-view and nixio to access the section.

What do you think happened there? Was it a nix issue or a sshfs issue?

Thanks.

Sincerely Ajay

jgrewe commented 8 years ago

ok, actually the hdf5 lib can do it if a certain flag is set. https://hdfgroup.org/wp/2015/04/parallel-io-why-how-and-where-to-hdf5/ but we do not support it at the moment.