search

jdibenes / hl2ss

HoloLens 2 Sensor Streaming. Real-time streaming of HoloLens 2 sensor data over WiFi. Research Mode and External USB-C A/V supported.
Other
219 stars 53 forks source link

Questions about Remote Unity Scene #78

Open knsjoon opened 11 months ago

knsjoon commented 11 months ago

Thank you very much for this repo available0. I appreciate it a lot. As I am a pure CV researcher without much knowledge about Unity and Hololens2, I would like to ask a few questions about Remote Unity Scene.

I am thinking of making a service or app that retrieves RGB-D (front camera of Hololens) and pose with your application and run YOLO V8 to detect objects and detect its size and position to draw a 3D bounding box around it on a computer. This has to be shown back to the HoloLens with the object type and the bounding box. I think Remote Unity Scene does just that by using eg. Create primitive.py or Create text.py.

My question is,

  1. I would like to add complicated .obj or .ply files also to the scene by python (like a car model with a lot of faces and vertices) or I would like to somehow pop up a internet brower next to an object of interest on HoloLens. Also, maybe I would like to have a forward and backward button to show different contents when users press those buttons (like one time, it shows bounding box and text for a bed and then next button pressed, it shows chair with its label and bounding box)

If I modify your code in some ways either on python side or C# side, would these functionalities be possible to be integrated into your work seamlessly or is it something that are related to appxbundle (therefore, unmodifiable) I would greatly appreciate your answers and helps.

jdibenes commented 11 months ago

Hello, Yes, I think it is possible to integrate this functionality without modifying the C++ code of the plugin and appxbundle. For example, the obj or ply and the bounding boxes could be sent to the Unity application using the message queue of the plugin or as a web request. Then, the model can be added to the scene using the OBJ Loader from the Unity Asset Store, and the bounding boxes and text can be displayed as you mentioned. The buttons can be created with MRTK. All of these features would only require changes to the Python and C# code, and the Unity application.

knsjoon commented 11 months ago

@jdibenes Thank you for your kind and informative answers. As I am very new to all these, I felt very lost these days and your guide shed a big light to it! Haha. I am less lost! Thank you once again!