Allow uploading and displaying arbitrary HDF5 stacks

catmaid / CATMAID

Collaborative Annotation Toolkit for Massive Amounts of Image Data

https://catmaid.org

GNU General Public License v3.0

188 stars 56 forks source link

Allow uploading and displaying arbitrary HDF5 stacks #1524

Open clbarnes opened 7 years ago

clbarnes commented 7 years ago

In order to display the results of synapse detection (and other such pixel labels)

[ ] Schema for HDF5 pixel label data, including offsets for block volumes, information on the image stack it's associated with(?), text associated with label values
[ ] Uploading, storing and serving HDF5 files
[ ] Editing these HDF5 files as more bits of the volume are filled in (by e.g. skeletonwise synapse detection)
[ ] Filtering to view particular labels in the UI

Issues:

[ ] Can only store 24-bit integers using current methods of passing data from HDF5 to PNG to webGL
[ ] Storing HDF5s in such a way that updates can be made efficiently, and they can be read by many people at once, may be tricky

clbarnes commented 7 years ago

For my own reference:

File uploading, see skeleton.import_skeleton()
HDF5 serving, see tile.py
WebGL filters, see pixi-layer.js (bottom of the file)

Thanks to @aschampion

clbarnes commented 7 years ago

HDF5 is, it turns out, really slow.

As a test, I made an zeroes array of int32s of size (22775 x 18326 x 462), and then added a 300x300 square to the first z plane which was a ring of 1s ("the detector has run over this area") around a block of 2s ("there is a synapse here"). Chunks of size (256, 256, 1), no explicit compression: file size is 1.6MB. It loads a chunk about every 500ms, which is barely usable.

@aschampion suggested looking into the python bindings for the Keller Lab Block format.

I think there's also an issue with the hdf5 tile source, as it's getting data nowhere near where my stack viewer is pointed.