neuronlistz - fast access to neuronlists stored as zip files

natverse / nat

NeuroAnatomy Toolbox: An R package for the (3D) visualisation and analysis of biological image data, especially tracings of single neurons.

https://natverse.org/nat/

64 stars 28 forks source link

neuronlistz - fast access to neuronlists stored as zip files #456

Closed jefferis closed 3 years ago

jefferis commented 3 years ago

neurons are best stored in the recent qs format

write.neurons(Cell07PNs[1:5], tf2, format='qs', include.data.frame = T)

You can also embed the metadata in the zip so that one file should contain everything.

For 3GB files, I get 20ms per record random access. Faster performance (order 5ms) could be obtained with https://github.com/jefferis/razip. Reading sequential groups of files is much faster.

jefferis commented 3 years ago

@alexanderbates first pass at storing neuronlists as efficient zip files for random access

coveralls commented 3 years ago

Coverage increased (+0.08%) to 81.396% when pulling 0077686b58e390cb03eb8df4f54164caded1f437 on feature/fast-nlzip into 7f665de62eadce4390fd690db5bea8ec02b71b60 on master.