Streaming Sensor Data

[task-master98]

Hi, Firstly thanks a lot for this work as this has immensely benefitted me. I am currently working on a Unity research project using the Hololens 2. Now my task is simple, I would like to stream the sensor data (grayscale cameras and the depth sensor) on a renderer such as the Quad object. I have been going through the hl2ss api but have been struggling with understanding the core logic. Can you tell me which functions I would need to call to simply get the sensor data so that I can stream it?

[jdibenes]

Hello, For your task, you would need a hl2ss Unity client to receive and decode the stream data, which we don't have. For Unity apps that don't run on the HoloLens, I can only suggest to take the hl2ss C++ client and make a Unity plugin with it. The functions that you would need to expose to Unity are mainly in the hl2ss multithreaded module (hl2ss_mt). These are the start(), status(), stop(), and get_packet() methods of hl2ss::mt::source. For each sensor you want to stream, create an instance of hl2ss::mt::source in the Unity plugin. Then create wrapper functions for the methods of each instance, which Unity can call (e.g., using extern "C" __declspec(dllexport) if on Windows). For an example of the hl2ss multithreaded module, see the hl2ss matlab client and main.cpp in the C++ client folder.

[task-master98]

Hi, I'd just like to be clear, the application would be running on Hololens. Essentially I would like to recreate something like this

You see this example they are broadcasting the data on a window which is a holographic object on the Hololens 2 device. I would essentially like to recreate this. I want to stream data from the depth sensor and the 4 grayscale cameras on separate windows like this. Please guide me for the same.

Thanks!

[jdibenes]

Ah, in that case this repo does that https://github.com/petergu684/HoloLens2-ResearchMode-Unity We also have a prototype in https://github.com/jdibenes/hl2da

[task-master98]

Hi so I am checking out the prototype you pointed out https://github.com/jdibenes/hl2da. However, I do not see any objects in the hierarchy, can you tell me which script is attached to which component? Are you streaming the data somewhere?

[jdibenes]

Open the SampleScene in the Scenes folder. The main script HoloLens2DA is attached to the Main Camera under the MixedRealityPlayspace object. SlateBlank contains the quads on which the streams are displayed.

[task-master98]

I understand now! Thanks a lot for your help!