search

bmegli / hardware-video-streaming

Hardware Video Streaming meta repository
Mozilla Public License 2.0
24 stars 6 forks source link

Applicability question #7

Closed realizator closed 4 years ago

realizator commented 4 years ago

Hello @bmegli, You mentioned you've streamed Realsense video to the Unity. We have a similar task, and need a low-latency livestream from the stereoscopic camera (StereoPi, based on Raspberry Pi) to the Oculus Quest. How do you think, can we use your solution? If it will have hardware-accelerated decode on Oculus Quest?

Regards, Eugene

upd> This ticket is opened following your answer here: https://answers.unity.com/questions/1627714/stream-live-camera-view-over-udp.html

bmegli commented 4 years ago

Hi @realizator,

I would start by examining ALVR which claims working (alpha) support for streaming to Oculus Quest. If not as end-to-end solution (PC vs RPi) then at least to get idea how to start.

The first question you should ask yourself is - do you really need hardware acceleration.

As far as HVS is concerned

Encoding

HVE

FFmpeg

Have a look at FFmpeg HWAccelIntro Platform API Availability:

In the context of RapberryPi you have:

Examining the table FFmpeg API Implementation Status:

This means that OMX would be the technology to go for hardware encoding on RPi.

I would first try with ready-made tools if hardware acceleration worked (gstreamer and FFmpeg).

Coming back to HVE

It is likely that OMX uses the some kind of interface that VAAPI does in FFmpeg and would require only minor changes in the code (see this issue).

Decoding

HVD

HVD supports multiple hardware decoding technologies but none of them is supported on Android.

FFmpeg

I know little about OculusQuest apart from the fact that it has Snapdragon (ARM) and runs Android.

My experience with FFmpeg hardware accelerated decoding on Android is bad (see this issue).

Unless something changed, based on my past experience, I would discourage you from using FFmpeg based MediaCodec hardware decoding on Android. Maybe I just have bad memories. Maybe it should be only used in zero-copy scenario on low-end Android devices (which doesn't seem to be possible with Unity)

ALVR

Which brings us back to ALVR, which as was stated claims support for streaming to OculusQuest.

Transport

Apart from hardware or software encoding/decoding you have to transport the data between the devices. Again, I would see what ALVR does as a starting point.

I have managed to run streaming pipeline to Unity application on Android but I would say that it didn't meet my expectations.

My NHVE, RNHVE and UNHVD:

You may also have a look at DJI Android Video Stream Decoding Sample, which was open source as far I remember and used hardware decoding through MediaCodec (FFmpeg for parsing H.264 and MediaCodec directly from Java as it was originally designed to be used, not through FFmpeg).

Kind regards

bmegli commented 4 years ago

Hi @realizator

If you have no more question I am going to close the issue.

realizator commented 4 years ago

Hi @bmegli, Thank you for your tips! We are now in the process of experimenting.