search

KitwareMedical / SlicerVirtualReality

A Slicer extension that enables user to interact with a Slicer scene using virtual reality.
Apache License 2.0
119 stars 58 forks source link

Integration of 3D Slicer Communication via OpenXR into Unity Project #177

Closed vincenzocivale closed 3 months ago

vincenzocivale commented 3 months ago

Hi,

I’ve been exploring the SlicerVirtualReality project and am particularly interested in its VR visualization and interaction capabilities. I am working on a project in Unity and would like to integrate communication with 3D Slicer using OpenXR.

I noticed that SlicerVirtualReality supports communication through Holographic Remoting and OpenXR. I’m wondering if it is possible to directly integrate this communication capability into a Unity project.

If anyone has experience with a similar integration or can offer guidance on how to proceed, any advice or examples would be greatly appreciated.

Thank you in advance for your help!

Best regards,
Vincenzo

lassoan commented 3 months ago

We used Unity for rendering in all HoloLens projects (exporting models from Slicer or sending models/transforms in real-time from Slicer). However, we can now do everything in Slicer - import all medical imaging data from DICOM, create surgical plan, connect to trackers/surgical navigation systems, register the patient, show surgical plan and tools in real-time on the HoloLens, including volume rendering, real-time warping transforms, etc. There are just a few things to iron out, such as laser for picking, display of Qt widgets, and binding of voice commands (some small fixes or developments are needed).

Unity feels very limited now,because all these features that are readily available via Slicer would take years to redevelop in Unity. If you have some useful features that you implemented in Unity that you would want to use in Slicer for the AR display then you can still exchange data in real-time between Slicer and Unity via OpenIGTLink (as it was done in https://github.com/BSEL-UC3M/HoloLens2and3DSlicer-PedicleScrewPlacementPlanning), but it may just not worth it. It may be easier to port those features to Python or C++ and run everything in Slicer.

vincenzocivale commented 3 months ago

Thank you for your detailed response (it's one of the most comprehensive I've received on an online forum).

I understand the limitations of Unity when it comes to developing features that software like 3D Slicer already provides. However, my goal is not to recreate these features in Unity but rather to stream the 3D model displayed in 3D Slicer into Unity in real-time.

Regarding the project that uses OpenIGTLink, I have examined their code, but a major limitation is the lack of synchronization of the 3D model. The model needs to be exported from 3D Slicer and then imported into Unity, which also necessitates reimplementing clipping functionality in Unity that is already handled by 3D Slicer.

I am not considering using their plugin because it does not support creating Mixed Reality UIs and AR functionalities. Additionally, I am looking to integrate the features of 3D Slicer with other plugins developed by our lab, which would be more manageable and versatile in Unity.

My objective is to visualize the 3D model rendered in 3D Slicer live within Unity. I was considering leveraging OpenXR streaming, similar to holographic remoting, to improve real-time updates of models and geometric information from 3D Slicer in Unity scene. However, I'm unsure about the feasibility of this approach.

Do you have any suggestions or ideas on how to achieve real-time streaming of 3D models from 3D Slicer into Unity? Any guidance or recommendations would be greatly appreciated.

[EDIT] After some research, I found that OpenIGTLink also supports sending polydata messages. I think I will try to implement the code to handle this type of message in order to enable the transmission of 3D models.

If you have any suggestions on how to do this or if anyone has already done it, let me know before I close the issue.