aschampion
commented
5 years ago Day 1 meeting outcomes
Strategy
- We'll build towards the draft plan for now, but may not implement painting/mutation in CATMAID initially if non-concurrent editing through Paintera is sufficient.
Design
- Multi-block locking is the responsibility of the application layer (in this case, the webservice) rather than of any N5 implementation.
ImageBlockCachewill eventually be replaced by an external library that can be shared with the webservice, so should be written such that it can be compatible with future features like multiscale and multilabel, even if it does not initially implement them.
Tasks
- [x] @bellonet and @axtimwalde will generate an example label N5 volumes from existing contour data.
- [x] @hanslovsky will demo Paintera with this dataset once it is ready.
- [x] @aschampion will make the dataset browsable in CATMAID via H2N5 once it is ready.
- [x] @aschampion will work on n5-wasm bundling and then the
ImageBlockLayerprototype.
This is a tracking ticket for MVP visualization and painting of label volumes via CATMAID.
Goals
Non-goals
Our vision for CATMAID does not include a rich, general-purpose painting tool or segmentation proofreading tool, encompassing features like:
Instead, we believe these are the purview of other tools like Paintera / BigCat / neuroglancer and that CATMAID should interoperate with these tools. Our vision for interoperation is incremental, progressive enhancement through three general stages of:
The plan below builds towards this vision. Relatedly, features that would be handled by such a state synchronization backend are also non-goals, such as the N-dimensional persistence itself, efficient delta exchange and conflict resolution.
Prior work
A painting layer was previously prototyped by Larry in CATSOP (#533, catsop feature branch). However, this was never merged to mainline CATSOP (which in turn was never merged into mainline CATMAID). Painting was never a development focus of CATSOP; the majority of CATSOP development was dealing either with SOPNET-specific issues, WebGL support (merged into CATMAID), database scaling optimization (partially merged into CATMAID), or the more general problem of dependent dataflow synchronization in a distributed, collaborative setting. None of these are goals here.
Draft plan
General prerequisite infrastructure
package.jsonand webpack bundle for frontendindex.htmlnumjsis built on top ofndarrayand has the most useful functionality, slicing and view.Volumetric stack access and rendering
ImageBlockLayer, comparable toTileLayer, for rendering N5 image volumes in theStackViewer. Note that unlike tile layers this will be WebGL-only (via Pixi for now) likePixiTileLayer, without a legacy DOM fallback.StackLayerabstract base class which bothTileLayerandImageBlockLayercan extend (or into mixin/component if appropriate)Option AImageBlockLayerfromPixiTileLayer, keeping the majority of tile creation logic.ImageBlockLayershould refer to anImageBlockCache(orImageBlockStoreorImageBlockManager), similar to the global texture cache.ImageBlockCacheshould have a CATMAID event bus for mutation and eviction, to whichImageBlockLayersubscribes, orImageBlockLayermust have explicit notification methods for block-specific redraws. This is necessary for block mutation during painting.ImageBlockCacheshould also track a state identifier for the last version of each block in the cache received from n5-wasm. Initially this will be either an mtime or content hash.imageValueAtLocationor similar method toTileLayerthat looks up the image intensity value for a location inside the current view boundary. This would already be useful for querying intensity in raw stacks, but will be necessary for label picking in label stacks.http://example.org/path/to/folder.n5/group/dataset/%SCALE_DATASET%/0_1_3Label stack specification and visualization
RaworLabel. @tomka and I have discussed alternatives (e.g., property ofProjectStack, specification via relation like stack groups and channels) and are fine with this being a new column directly onStack.StackClassInstancerelated properties or in the N5attributes.json. For now this is stored in a magicStack.metadataproperty,catmaidLabelMeta.specialLabels.SkeletonAnnotationsin that they coordinate the selection state, display and mutation of label annotations in tools, but rather than having one per project, there is one per open label stack.ImageBlockLayeris supportedImageBlockCacheand theImageBlockLayer, possibly through a wrapper. In this design label mapping occurs in Javascript and the texture uploaded to the GPU would be the global label IDs.LabelFilterfilters that are applied to theImageBlockLayerthrough the existingPixiLayerfilter system. This is already partially implemented (cc @clbarnes).RandomizedLayerColorMaporObjectLabelColorMapfilters).ObjectLabelColorMap, but can later be adapted to assign particular colors via a label selection source model just as skeleton sources do now.Painting
ImageBlockLayer.ImageBlockCachebacking the label layer.Option BStrokes are painted directly into theImageBlockCache.N5 doesn't support multi-block locking, but rust-n5 could be trivially extended to do so.The webservice itself will handle multi-block locking, as it would also be responsible for any future, more elaborate, conflict resolution mechanism.ClassInstancerepresenting the CATMAID ontology object for this label.Unresolved questions
Alternatives
ImageBlockCachefunctionality in wasm. This functionality will likely be provided by an external library in future iterations anyway.N5Writerdirectly over WebDAV and bypass the CATMAID backend for block writing, but WebDAV is a mess and few implementations provide locking.N5Writerfor a generic RESTful HTTP backend.Known limitations
Misc. notes
cc @tomka @bellonet