search

plotly / dash-slicer

A volume slicer for Dash
https://dash.plotly.com/slicer
MIT License
24 stars 8 forks source link

API to specify overlay data #17

Closed almarklein closed 3 years ago

almarklein commented 3 years ago

The slicer needs to be able to overlay segmentation data (e.g. masks). Two suggestions:

slicer = VolumeSlice(volume, mask, ...)

# Suggestion 1: provide a mask shaped LxMxNx4 (it includes an alpha channel)
slicer.setOverlay(mask)

# Suggestion 2: provide a mask shaped LxMxN plus a colormap. 
slicer.setOverlay(mask, colormap=[(0, 0, 0, 0), (255, 0, 0, 100)])

With 1) the user has somewhat more flexibility, but the user would have to compose the mask, and the memory cost is high. With 2) we can still support multiple "layers", by allowing the mask to be integer, we can provide a default colormap or allow providing a single color to make single-layer masks easier.

Maybe there are other ideas?

emmanuelle commented 3 years ago

Thanks for the issue. What would be passed to Javascript for mask would be b64 image strings I guess? Therefore the memory cost is the same whether the image string is created from a multichannel or single-channel array.

emmanuelle commented 3 years ago

in terms of user-facing API probably 2) will be a more common case I guess. I'd vote for 2).

emmanuelle commented 3 years ago

@surchs @nicholas-esterer please chime in!

almarklein commented 3 years ago

What would be passed to Javascript for mask would be b64 image strings I guess?

For now, yes, but I'm hoping that we can keep this an implementation detail, so it can be changed without affecting the public API.

almarklein commented 3 years ago

Proposal pr: #18

nicholas-esterer commented 3 years ago

Would masks have to be the same size as the volume already in the slicer? Or could we specify an offset to upload smaller masks?

almarklein commented 3 years ago

In the proposed PR the mask has to match the shape of the volume. But we could allow for an offset, if we want. I can imagine that being handy to be more memory efficient ...

almarklein commented 3 years ago

Opened PR for the covid app, to show what it currently looks like: https://github.com/plotly/dash-sample-apps/pull/525

emmanuelle commented 3 years ago

Would masks have to be the same size as the volume already in the slicer? Or could we specify an offset to upload smaller masks?

It'd be worth checking the performance burden of passing empty arrays for "uninteresting" parts of the mask. From what I remember from your tests @nicholas-esterer, the resulting b64 image strings are tiny but it still takes some time to do the serialization. With the current approach of passing not all the data to the client but only what is needed (plus some caching) this could be different now...

almarklein commented 3 years ago

We could also test, per slice, whether the overlay is all zeros in the alpha channel, and just send null in that case, skipping the encoding step.