search

jdibenes / hl2ss

HoloLens 2 Sensor Streaming. Real-time streaming of HoloLens 2 sensor data over WiFi. Research Mode and External USB-C A/V supported.
Other
217 stars 53 forks source link

Simultaneous capturing of video locally on HL2 while streaming #143

Open anjelomerte opened 1 month ago

anjelomerte commented 1 month ago

Hey Juan Carlos, thanks for engaging with the community on your lovely project! Since a lot has changed since the last time I worked with your project, I got two things on my mind:

  1. Is it possible to record video footage locally onto HL2 while streaming it to an external PC?
  2. For a server-controlled streaming workflow, is it still the best way to poll for server input or are there any new possibilities I am not aware of?

Best regards!

jdibenes commented 1 month ago

Hi,

  1. No. Support for that would have to be implemented in the C++ code.
  2. Yes, streaming is still controlled by the client.
anjelomerte commented 1 month ago

Regarding 1, is it simply not possible with the hl2ss plugin or is it inherently not possible to record while also stream? In other words, if I were to implement video recording in Unity/C# while the hl2ss streaming plugin is running, would that cause problems? If I had a HL2 with me I would simply test it but currently I don't, that's why I am asking.

jdibenes commented 1 month ago

It should be possible to record while streaming with some caveats. For example, if the hl2ss plugin is streaming the PV camera then the Unity/C# code must access the camera in shared mode. Otherwise, the Unity/C# code will take over the camera and streaming will stop (the hl2ss client will be disconnected). Conversely, if the Unity/C# code is using the camera and then some hl2ss client requests the camera stream it must do so in shared mode. Otherwise, hl2ss will take control of the camera and the Unity/C# stream will be closed. So some synchronization between server and client is necessary. For example, the Unity/C# code could initialize the camera to the desired resolution and framerate and keep the camera running for the duration of the app, while the hl2ss client always connects in shared mode (in shared mode, resolution and framerate cannot be changed).

anjelomerte commented 1 month ago

Thank you for your input! Your proposed solution sounds feasible. So would you agree that this is the best approach to preview captured video footage on the HL2? I mean capturing by the client is possible but I see no straightforward way of getting that captured footage back onto the HL2 or am I missing sth?

jdibenes commented 1 month ago

Yes, I think so. But just to preview the video captured by the client, I would add callbacks to the C++ code to detect when a client connects/disconnects and forward raw frames to the C# code to avoid managing two streams.

anjelomerte commented 1 month ago

You are talking about viewing the video stream in real-time on the server right? What I want to do is capture let's say 1min of the stream and then review that captured 1min footage afterwards. If this is also possible by forwarding frames I am grateful for any further details on how that can be accomplished.

jdibenes commented 1 month ago

You are talking about viewing the video stream in real-time on the server right?

Yes, on the HoloLens display.

What I want to do is capture let's say 1min of the stream and then review that captured 1min footage afterwards. If this is also possible by forwarding frames I am grateful for any further details on how that can be accomplished.

I think it can be done in this way:

  1. Create additional C++ functions (and export them using PLUGIN_EXPORT) and global variables to register C# callbacks for client connected, disconnected, and frame capture. For the client connected and disconnected events, the C# callbacks could be called (in the C++ code) before and after PV_Stream (the one after accept) respectively. The frame capture C# callback could be called inside the PV_OnVideoFrameArrived function. The captured video frame could be passed to the frame capture callback, for example, as frame_capture(IntPtr frame, ulong timestamp).
  2. The C# callback for client connected creates a new file or folder for recording (in the C# code) and the callback for client disconnected finalizes the recording (e.g., closes the file). The frame capture callback adds the received video frame to the file/folder. For recording a subset of the video instead of the entire duration of the client connection, one could look at the timestamps of the video frames to determine time elapsed.
  3. Finally the recorded video frames are loaded (in the C# code) and shown to the user.
anjelomerte commented 1 week ago

Thank you, I will dig into the various approaches and get back