Segmentation support in MoBIE BVV integration

[tischi]

Hi @ekatrukha,

Starting an issue about showing meshes in BVV from MoBIE.

The current mesh code is all here:

https://github.com/mobie/mobie-viewer-fiji/tree/main/src/main/java/org/embl/mobie/lib/volume

Here is where a specific mesh is added to the current volume viewer:

https://github.com/mobie/mobie-viewer-fiji/blob/5d95facf26350d278b787049fe2480f3cc7f3090/src/main/java/org/embl/mobie/lib/volume/SegmentVolumeViewer.java#L283

I guess converting the current mesh into an imagej-mesh should not be a big deal. I can also help looking into this. Let me know if I should have a look!

[ekatrukha]

Hello @tischi,

before we start digging. Are meshes loaded from the disk, where they are stored in some format (.stl)? Or do you generate them in Mobie from some thresholded source, programmatically? Or both?

[tischi]

The are created on the fly; they are not stored on disk.

[tischi]

I slightly widened the title of the issue. For me it would also be interesting to check how a label mask image volume rendered with Glasbey LUT would look like; maximum projection would likely not be useful for label masks :-)

[ekatrukha]

Max proj no, but 'volumetric' should be ok. You just need to narrow down alpha value so everything is not transparent, but keep LUT wide.

[tischi]

Is the alpha value adjustable?

I would add the "normal" volume rendering as an option for the label masks to our branch such that we can test this, ok?

Is there any way that I could test the Glasbey LUT from within my IntelliJ IDE? I think there was some trick to "link in" the Fiji folder, but I am not sure...

[ekatrukha]

It is adjustable, check the readme of bvv-playground.

There is a method to load IndexColorModel in the Converter setup, so you just need to read glabsbey values into it from disk (ImageJ LUT) or somewhere else.

I can check it for you tomorrow.

[tischi]

There is a method to load IndexColorModel in the Converter setup, so you just need to read glabsbey values into it from disk (ImageJ LUT) or somewhere else. I can check it for you tomorrow.

Thanks! That would be very helpful for the testing.

[tischi]

I added code to display label mask images (aka segmentations) with BVV. However, the "classic example", which are the cells from the platynereis dataset are of Long datatype, which throws an error:

Cannot display cells in BVV, incompatible data type:
net.imglib2.type.numeric.integer.UnsignedLongType

Is that expected? Could be that there is no support in SpimData... Would BDV be able to display this at all or would be need to convert to something else? I think you mentioned that BVV can only do uint16, is it?

[ekatrukha]

For cached multires it is only UnsignedShort, uint16, indeed. I can add loading of Long either truncated to max of 65535 or 'cyclic', i.e. reminder of division by this max.

[tischi]

That would be great! Maybe cyclic would be best such that segments >65535 would still be rendered with different colours.

[ekatrukha]

Hello @tischi,

I've made Glasbey LUT version and UnsignedLong data loading. In the end the cycling enough to do for 256, since it is the range of the LUT. You can check results and play with it a bit here. I've made two options, one with "dark render" (version 1) and one with clipped volume (version 2).

This is version 1 view

But main conclusion to me is that it seems like the multires image is not the best way to show segmentation results in 3D. The reason behind it is that scaling by factor of 2 scrambles all the labels in 3D (average intensity becomes weird). That leads to the "scrambled" segmentation. See version 2 initial view And after it loaded a bit better resolution

The areas of fine, thin labels (with thickness below the current optimal/displayed resolution) are getting scrambled. It happens also in BDV, but it is less noticeable, because higher resolution can be loaded any time and fast in one plane. For BVV to have fine details correctly one would need to load it all to the GPU.

Well, you can check the result and play with it yourself. So I guess meshes would be a solution. How many objects do you have in this segmentation? Some time later we can try to implement meshes generation and loading.

[tischi]

Not sure if that information helps, maybe you know all of this already: The resolution pyramid for the labels was created using a nearest neighbour / sampling strategy, thus there should be no averaging of label values. In BDV, for the display one has to also use nearest neighbour interpolation (this can be configured, pressing the I key). In fact, I think my code in MoBIE prevents label masks from ever being interpolated, even if the user is pressing the I key.

The reason behind it is that scaling by factor of 2 scrambles all the labels in 3D

Where does that factor 2 scaling happen in BVV? Could one configure it to do something else than averaging? For instance, taking a random sample?

How many objects do you have in this segmentation?

Around 16000 I think

So I guess meshes would be a solution.

Yes, that's why so far in MoBIE I am using meshes for the display of segmentations. However, only (the few) segments that are actively selected by the user are rendered, because creating 16000 meshes on the fly would be too slow and I am not sure whether the 3D Image Viewer could handle it. But I don't think I ever really tried to benchmark/push this... Anyway, I think a good start would be to just reproducing my current 3D Viewer Mesh implementation with BVV.

Well, you can check the result and play with it yourself.

Will do.

[tischi]

I can reproduce the scrambling of values at the borders of the labels, but don't really understand why that happens. Do you have a reference for how the volume rendering algorithm works? Does it just take the first non-zero value that it finds along a ray?

[tischi]

Version 1 is better I think, one can get to Version 2 manually by "moving into the sample".

[ekatrukha]

If the pyramid was built using nearest neighbor, than I guess my hypothesis was wrong and it is not a culprit.

Do you have a reference for how the volume rendering algorithm works?

Not in a written form, no, it is unpublished. I know from reading/tinkering with the code and conversations with Tobias :) In principle, volume rendering in BVV is optimized for the speed and it makes some assumptions about data. We can remove those assumptions and see if the quality of the picture improves. So there are a few possible suspects that we can test: 1) First one is dithering (explained here). I will try to remove it and see if it is a problem. 2) The second one is a variable step along the ray. Each screen pixel shoots a ray through the volume and sample/accumulates max (for max intensity) or "alpha blended" intensity values. In the current form the step size varies: BVV takes smaller steps on the part of the ray closer to the camera and makes larger steps further away. In many other 3D renderers (sciview) the step size is constant to avoid artifacts. That we can also change and see if the scrambling goes away. 3) There is some bug in my conversion of UnsignedLong. I need to think about it, so far it looks ok. 4) Something else.

Does it just take the first non-zero value that it finds along a ray?

Kind of, not really. It is making alpha blending of accumulated voxels along the ray in the shaders . It should stop when alpha value is more than 1. But! If we put alpha range in the ConverterSetup of BVV to 0-1, it should stop at the first sampled voxel, so yes (unless I miss something). But then we have point 2) from above.

I am going investigate a bit.

manually by "moving into the sample".

Yeah, this is why I think that future bvv-minimal BVVBrowser should have a clipping controls.