Add interactive Qt panel to VR scene

[cpinter]

A very flexible way for showing menus in VR would be to implement off-line rendering of any Qt widget in a texture that is shown on a VTK plane actor, which can respond to virtual mouse events generated by the laser pointer (currently called probe mode in VTK) in VR.

Tasks that need to be done, start with first and only move to next one once it is done.

• [x] Generate 2D texture with Qt of arbitrary QWidget. Done when you have a QImage or QPixmap containing a QWidget with some other widgets in it (for example a QFrame with a QVBoxLayout having a QCheckBox and a QSlider on it) so that the widget is never shown on the Windows desktop (i.e. everything happens in the background).
• [x] Show texture generated in the previous step on a VTK plane actor (for example vtkImageActor). Done when the Qt GUI is visible in the scene shown in VR (positioning can be arbitrary at this point).
• [x] Refactor VR interactor style according the new events design (Csaba)
• [x] Re-add laser pointer (called probe mode in vtkOpenVRMenu*), TBD
• [x] Send laser pointer events to Qt, re-render the widget if necessary, TBD
• [ ] Create default "pipboy" style panel with basic settings. Show up attached to left forearm if left controller is turned outside 180(ish) degrees. Make panel look like it's obviously interactive (back side not visible, slight frame on the edges, etc.)

Useful UI panels (extend as needs arise):

• [x] "Main menu" widget that takes the user to the "apps" (below)
• [x] Subject hierarchy (show/hide)
• [x] Segment Editor (simplified)
• [ ] Interaction settings: group objects (moving them under the same interaction transform), toggle selectable, etc.

Related issues

Related pull requests

• https://github.com/KitwareMedical/SlicerVirtualReality/pull/100

[pieper]

💯 this will be a great feature!

[jcfr]

To follow up on this, @martinken recently added classes to VTK for adding Qt widget into the VR scene:

• vtkQWidgetRepresentation.h
• vtkQWidgetTexture.h
• vtkQWidgetWidget.h

[cpinter]

@jcfr Perfect! @mohammadrashid0917 just started working on this. We'll re-adjust the plan to use these classes. Mohammad, please hang on. In the meantime you can look at these classes.

[lassoan]

@jcfr We were lucky this time that minimal/no work was duplicated, but we need to coordinate a bit more closely to make sure we avoid parallel efforts. How could we do that? Where can we find planned VTK virtual reality developments? Do you have a category in the VTK bugtracker, labs page, …?

[jcfr]

How could we do that?

The plan described at the top of this issue is outstanding. I should probably have done a better job communicating about it with folks working on OpenVR at Kitware.

Moving forward, I think sharing the plan like the one at the top of this issue on VTK discourse would help.

Internally, the folks working on OpenVR features know that I am coordinating effort on the Slicer side and I think it worked pretty well as I was able to follow up here.

That said, I agreed that the feedback loop could be shorter.

May be having a list of contact associated with key VTK area would help ?

Where can we find planned VTK virtual reality developments?

Considering that this type of feature are driven by customer needs and often planned and executed in a short time frame, i think it is challenging for us to communicate about this features with a high level of granularity and far ahead.

[jcfr]

To follow up, here is a Slicer PR backporting the specific VTK changes to Slicer/VTK fork. See https://github.com/Slicer/Slicer/pull/1141

I have checked that it compiles on Linux, after someone check that Slicer/VTK compiles on Windows and macOS, it will be integrated.

[cpinter]

We have made some good progress on the GUI widget and the laser pointer. Basically we created a new GUI widget markup that uses a different way to render the Qt widget on the VTK plane as texture (not the shared context as it is in VTK, which crashed when used in Slicer), and we added a little test module that creates a "laser pointer" (just a line) the intersection of which to the Qt widget is calculated and events generated. So as the last larger step we need to support a laser pointer usable from VR with the controller.

I've recently seen some news from Paraview in this topic and thought we could use some bits and pieces from there if compatible. Does anyone has personal experience with VR in Paraview and maybe also with the in-VR GUI widget feature in Paraview?

[adamrankin]

@cpinter do you have any links to the paraview pieces/issues/PRs?

[cpinter]

I don't. I'd just search for usage of vtkQWidgetWidget. Although it doesn't work in Slicer because the context sharing method is somehow not compatible, we could see how the laser pointer is implemented etc.

On Wed, Jan 19, 2022, 21:09 Adam Rankin @.***> wrote:

@cpinter https://github.com/cpinter do you have any links to the paraview pieces/issues/PRs?

— Reply to this email directly, view it on GitHub https://github.com/KitwareMedical/SlicerVirtualReality/issues/43#issuecomment-1016871048, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKDXHEMSUIDXDTMBJ724CDUW4SCNANCNFSM4FYZR7QA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[cpinter]

Using the branch https://github.com/cpinter/SlicerVirtualReality/tree/gui-widget-20230612 the interactions can be set up to work like shown in this video:

https://github.com/KitwareMedical/SlicerVirtualReality/assets/1325980/851adfd5-ae03-441d-8570-7da7b2b644a6

Note that I had to apply a few workwrounds in the code and use a bunch of scripting to achieve this. Next steps can be:

• [ ] Laser pointer is set up from the PointerSimulator Python module. Need to move the setup and the controller transform setting code to GUIWidgets
• [ ] Bind the menu button to show/hide the widget
  • [ ] When shown, bind the pick event to the trigger, otherwise use the default grab&move
• [ ] Make the GUIWidget markup node respect the parent transform
• [ ] Add handle to the widget so that it can be moved around (need to make it work with the above task)
• [ ] Separate press and release events to be able to drag a slider
• [ ] Refactor code so that GUIWidgets and VirtualReality modules are not so entangled
  • [ ] See handling of Pick event, which currently simply invokes an event, but it should perform the click instead
• [ ] Remove workarounds applied to make the interactions work (see vtkSlicerQWidgetRepresentation::OnTextureModified)