Basic implementation of monolithic DB1 format for neuronlistfh

natverse / nat

NeuroAnatomy Toolbox: An R package for the (3D) visualisation and analysis of biological image data, especially tracings of single neurons.

https://natverse.org/nat/

64 stars 29 forks source link

Basic implementation of monolithic DB1 format for neuronlistfh #453

Closed jefferis closed 3 years ago

jefferis commented 3 years ago

with tests

coveralls commented 3 years ago

Coverage increased (+0.0006%) to 81.312% when pulling d8e0cd80582ab98193dfa45722177a6d97158dc9 on feature/neuronlistfh-db1 into eae539fedd3c71315e290f63ae3f78a7d1300f29 on master.

jefferis commented 3 years ago

@alexanderbates let me know when you have tried this branch in anger.

jefferis commented 3 years ago

@alexanderbates I added some extra documentation in 810e9942a0bc20b6047ef5c9c8f3ef3c4e9f2a7c as well as checks that both the metadata and data files were present after saving? Could you just confirm that looks ok to you? I've made you a reviewer of this PR, so you can go ahead and merge if you're satisfied.

alexanderbates commented 3 years ago

@jefferis One issue I noticed is that sometime after the function has finished, a ___LOCK file for the datafile still exists. I am using it to save thousands of neurons on flyem1 and I am not sure what causes this behaviour. As far as I could see, the code did not otherwise error and I did not run things in parallel. I had to add in code to remove the LOCK file, when there is one, if I wished to save again.

jefferis commented 3 years ago

Hmm it is obviously supposed to delete the lock file when it finishes. Here is the code that should call it (using on.exit):

https://github.com/rdpeng/filehash/blob/07d9e9061daf9d69846ec2574ff0161f034d75d7/R/filehash-DB1.R#L274-L285

Do you have any indication that the insert is failing to complete?

alexanderbates commented 3 years ago

I think when the lock file is left, something has errored, and the datafile is not updated as I expected. I have not tested much though, because I only encountered this error a few times and both times did not investigate it thoroughly. I have not had it again since commenting here.