trisurf / surface opacity?

[ndawe]

Is it technically possible to change the opacity of a mesh? The API doesn't seem to allow it now but would it be easy to add?

[maartenbreddels]

No it's not gonna be easy. Rendering with transparancy requires special rendering tricks, and with volume rendering even more. It is a planned feature though, but help would be welcome 😉 .

[choldgraf]

@maartenbreddels I believe that transparency works with pythreejs, yeah? Maybe it'd be straightforward to figure out whatever solution they've got and port it here?

[maartenbreddels]

I doubt it, threejs does not support it out of the box, it is a non-trivial problem since rendering order matters.

[choldgraf]

sounds good - we'll try to figure something out w/ color in the meantime I guess

[maartenbreddels]

What can work, is ordering (I've seen some discussions on this), but that won't work for how I render trimeshes. I know a technique that is more general, and may work for everything except volume rendering, but VTK managed to solve it, so itI should be possible. It's really high on my wishlist, so it is gonna happen for sure!

[choldgraf]

I believe that @larsoner has worked with 3D viz stuff (and he is an old frenemy of VTK I believe). Maybe he has thoughts on this.

[larsoner]

Take all of this with a grain of salt because it's been a while...

Rendering with transparancy requires special rendering tricks, and with volume rendering even more.

If you want to do it correctly, yes it's a lot of work. But for a single trisurf like two brain hemispheres (our primary usecase over at MNE), naive transparency with backface culling and setting of blending modes is a good first-order solution (and would be necessary for a final version anyway). I have to imagine this is already exposed in three.js as it's pretty basic stuff.

IIRC VTK uses depth peeling, and I've never been able to enable it in mayavi in Python (how I interface with vtk) so I don't know how good it is.

The best solution AFAIK is the "weighted order independent transparency" two-pass rendering technique. There is discussion here and here with a working demo in this fairly readable gist. I don't know for sure if it works with volume rendering, but I think it's supposed to be general enough so my guess is that it does. This would need to be done at the three.js level, but presumably that's true of all of these more advanced methods.

[maartenbreddels]

Yes, the weighted order method is what I had in mind as well, I'm not sure indeed if it would work with volume rendering, but maybe a good first good step, and see from there (If I remember correctly, VTK also uses that, so should be possible. Many thanks for the info and the links.

[larsoner]

If I remember correctly, VTK also uses that

I looked a while ago and couldn't find any VTK reference to the algorithm.

So do you think a good first step is to expose at least front/backface culling for transparency?

Also, trisurf doesn't set the normals so surfaces look blotchy. WDYT about adding that?

I might have time to look at one or both of these things, but maybe not for some weeks.

[maartenbreddels]

So do you think a good first step is to expose at least front/backface culling for transparency?

It is exposed if that is what you mean.

Also, trisurf doesn't set the normals so surfaces look blotchy. WDYT about adding that?

Yes, should be added indeed, would make it look much nicer.

I might have time to look at one or both of these things, but maybe not for some weeks.

Awesome, I'm in the same situation, if you arrive at it sooner, feel free to ask me about the codebase etc, maybe on gitter in you think that can help. My gut feeling tell me that weighted order will not work with volume rendering, and you'd need depth peeling. Maybe if it is not too much work both methods should be supported.

[larsoner]

I'm confused, then -- @choldgraf why can't we get basic transparency working already?

[maartenbreddels]

First the alpha blending needs to be enabled during rendering (I could put this in). And also the faces need to be sorted in order to render back to front right?

[larsoner]

Ideally they would be sorted but as a first pass might not be necessary for our use case (our meshes are usually inflated / sphere-like so the culling may already do an okay job) but yes alpha blending is needed.

BTW people have made some progress on OIT:

https://github.com/mrdoob/three.js/issues/4814, https://github.com/mrdoob/three.js/issues/9977

And there are lots of three.js issues related to that to read if interested.

[larsoner]

It looks like we might want the renderOrder stuff exposed if it's not already, though:

https://stackoverflow.com/questions/15514274/three-js-how-to-control-rendering-order

[larsoner]

(Because we do have typically an opaque cortical surface on top of which we want to overlay translucent, color-mapped values)

[maartenbreddels]

Great, at least we have reference implementation/proof of concepts. I think threejs will only sort objects, not faces, since you actually don't know what the shader will do, so it probably relies on the center/boundingbox as exposed by the Object3d. But it's worth a try to have control over the alpha blending, and see what it looks like. I'm afraid people expect all of this to work out of the box, so I'd like to keep it a bit 'hidden' until we find a solid implementation..

[choldgraf]

Any movement on this? MNE Python is working on getting a GSOC student to build in some brain plotting functionality, but for a particular subset of data, plots like this would be super useful:

(basically a 3D mesh w/ some scatter points inside of it that you can only visualize if the mesh has a very low alpha)

[maartenbreddels]

There are some movements on this, the volume rendering is going through a refactoring, which will enable the OIT (order independent transparency), it is a difficult problem though.

[pkienzle]

Thanks for the hint. Backside culling does indeed make things look prettier:

import ipyvolume as ipv
import numpy as np
from numpy import sin, cos, pi

R, r = 1.0, 0.2
u = np.linspace(0, 2*pi, 100) # tube
v = np.linspace(0, 2*pi, 100) # ring
U, V = np.meshgrid(u, v)
x = (R + r*cos(U))*cos(V)
y = (R + r*cos(U))*sin(V)
z = r*sin(U)
fig = ipv.figure()
ipv.style.box_off()
ipv.style.axes_off()
ipv.zlim(-1, 1)
obj = ipv.plot_surface(x, y, z)
other = ipv.plot_surface(x, y, z+0.5)
obj.color = [0., 1., 0., 0.9]
obj.material.transparent = True
obj.material.side = "FrontSide"
ipv.show()

[GuillaumeFavelier]

But if I slightly modify the above script to add a third layer also with transparency, we obtain the following:

import ipyvolume as ipv
import numpy as np
from numpy import sin, cos, pi

R, r = 1.0, 0.2
u = np.linspace(0, 2*pi, 100) # tube
v = np.linspace(0, 2*pi, 100) # ring
U, V = np.meshgrid(u, v)
x = (R + r*cos(U))*cos(V)
y = (R + r*cos(U))*sin(V)
z = r*sin(U)
fig = ipv.figure()
ipv.style.box_off()
ipv.style.axes_off()
ipv.zlim(-1, 1)
obj1 = ipv.plot_surface(x, y, z)
obj2 = ipv.plot_surface(x, y, z+0.5)
obj3 = ipv.plot_surface(x, y, z+1.0)

obj1.color = [0., 0., 1., 0.5]
obj1.material.transparent = True
obj1.material.side = "FrontSide"

obj2.color = [0., 1., 0., 0.7]
obj2.material.transparent = True
obj2.material.side = "FrontSide"

obj3.color = [1., 0., 0., 1.0]

ipv.show()

I read https://github.com/mrdoob/three.js/issues/4814 and tested https://raw.githack.com/arose/three.js/oit/examples/webgl_oit.html and order-independant transparency looks amazing! @maartenbreddels how is the work going for the integration of OIT?

Work has started in mne-python to use ipyvolume as a 3D backend and having OIT would be perfect for us.