Q: Use SceneGraph instead of `gloo` + visuals

[larsoner]

Here is an example

https://gist.github.com/larsoner/25fa656c6d6e0b02b56c40a571bfb77c

Might make life easier, might not.

[EtienneCmb]

Hi @larsoner ,

Thx for sharing your gist !

I tried to used visuals.Mesh but there's two major issues :

• Light is not updated with camera rotation + weird specular light
• Depth issue with rotation because multiple mesh are superposed (z-sorting?)

And I'm not sure that with mesh superposition we could be able to reproduce this type of conjunction map?

I agree that gloo is not really user friendly (at all), but present some advantages :

• Light is correctly adapted with rotations
• No depth issue
• vertices / faces / normals are used only once and overlays are computed using a single 2D texture

[jasmainak]

@larsoner have you been spying on us or is it the GSS (gramfort secret service)? :)

[larsoner]

GSS :)

• The light updating is a known issue in VisPy (with a possible hack-ish workaround IIRC) that should be solvable pretty easily in principle with the flexible shader system.

• I'm not sure what you mean about the specular bits, it looks okay in my gist IIRC. (Again if it's broken, it should be fixable, though -- the math you do could / should be ported to Mesh if it's wrong!)

• I think the conjunction map might already work in my gist, as I plot the underlying brain first, then the activation on top as separate Mesh objects (a lot like in PySurfer/VTK). I think you could stack other activations on top, too, no custom shader coding necessary, but it would require testing to be sure. But in any case, it should be a solvable problem with multiple GL objects / draws because other packages do it (three.js, VTK, etc.) that way IIRC. Doing it with custom gloo shaders with multiple varying values or adding multiple cmapped RGB values in the CPU land to upload them to the GPU (if you use either of those approaches) don't seem ultimately sustainable based on the multitude of things people might want to plot, and how they might want to plot them. IIRC there is the polygon_offset in OpenGL that might work here, but admittedly I am out of my depth of knowledge on that one -- I tried to get it to work a little bit in my gist but gave up once the order-of-rendering method worked.

Basically what you have described so far are minor problems that must be solved at some level -- solving them at the SceneGraph level confers more benefits automagically (and to way more people, but this might be a secondary concern) than solving them manually at the gloo level with custom shaders. So it's worth thinking a bit before you get too far down the road on the gloo path to see if the SceneGraph + improvements path is more sustainable or workable.

Just my 2c, though, YMMV!

[jasmainak]

@larsoner the gist you shared does not really work for me (I'm using the environment.yml from MNE and master branch of vispy) and I get a frozen window. I seem to be stumbling upon this issue every now and again. We spent 2 hours with @EtienneCmb today to debug a frozen window issue like this one.

But yeah, I totally agree with you that simple fixes in vispy should be fixed using pull requests so that the fixes benefit a broader community and we're not reinventing the wheel.

What we have so far seems to work fine with not too much code, but I agree it could perhaps be even better to use high-level API.

[larsoner]

I get a frozen window. I seem to be stumbling upon this issue every now and again. We spent 2 hours with @EtienneCmb today to debug a frozen window issue like this one.

This is a problem with the vispy app system, so I'm not sure of its relevance for the discussion of whether to use gloo or scene...?

Light is correctly adapted with rotations

This is now fixed (by a variant of the suggested hack) in my gist, which is slow but works. Eventually we could update the Mesh shaders to speed it up.

No depth issue

I finally figured out that this is fixable with polygon_offset=(-1., -1.), polygon_offset_fill=True. Other layers can be added with larger offests, which I now demonstrate by adding a translucent green label with polygon_offset=(-2, -2) and just a handful more lines:

vertices / faces / normals are used only once and overlays are computed using a single 2D texture

I hadn't thought about this possibility, perhaps this could be more efficient if the number of Visuals added gets too large, which is a known bottleneck. :( One fix is to make Collections work properly in VisPy, but this would be a lot more work.

So maybe your approach is better, not sure... or maybe your texture-based approach could be integrated into Mesh in a way that still allowed sub-views, etc. to be used.

[jasmainak]

This is a problem with the vispy app system, so I'm not sure of its relevance for the discussion of whether to use gloo or scene...?

The only reason I brought it up is because what we have currently just works on my computer, but the gist didn't. So, I'm not sure if it's a generic vispy issue or just a problem with certain parts of the vispy code ...

[larsoner]

Try removing the app.use_app line, that's really the only thing that could/should be different

[jasmainak]

hmm ... nopes. In ipython, it freezes and from the terminal, the window doesn't stay open -- regardless of weather the app.use line is commented out or not. Visbrain-mne works for me from the terminal but I have the same freezing issue on ipython. Never mind, I'll try to figure out with Etienne, it's probably not so relevant to the discussion.

[larsoner]

Maybe you need a canvas.app.run(), which will start the event loop (and maybe put the app.use_app('pyqt5') back at the top)? Normally this is hacked by matplotlib, ipython, and vispy to auto-update when no user input is given, but it seems like your IPython might not be configured properly to do this.

[larsoner]

@EtienneCmb FYI if the texture-based approach does end up being the way to go for speed, there is a also a PR for this in VisPy

https://github.com/vispy/vispy/pull/1444

[EtienneCmb]

@jasmainak may be you should put a app.run() at the end of the file?

@larsoner, 100% agree with you that light improvements should be pushed to vispy. I was playing with polygon_offset this morning but couldn't find a way to make it works. But you did, awesome :+1: I was also reading this this morning but I don't really understand. Can you explain why each time you add a mesh, offset should be decreased? About rotations, I can see that indeed light is updated but not sure it's render correctly.

[jasmainak]

Maybe you need a canvas.app.run(), which will start the event loop (and maybe put the app.use_app('pyqt5') back at the top)?

Yes! It worked finally -- at least from the terminal. (Doesn't still work from ipython, but for both visbrain-mne and gist) It looks quite slow to me indeed but sounds exciting!

[larsoner]

The negative means "move out of the monitor", so by setting the polygon_offset more negative, then you make it closer to the viewer i.e. like what a higher z-order does it matplotlib