Support HW encoding and decoding on HL2

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience

https://microsoft.github.io/MixedReality-WebRTC/

MIT License

908 stars 282 forks source link

Support HW encoding and decoding on HL2 #27

Open grbury opened 5 years ago

grbury commented 5 years ago

Do we get performance improvements on HL2 by leveraging HW encoding and decoding? Would be great to know whats possible.

djee-ms commented 5 years ago

This is planned for the short term.

Note that there is already some preliminary support for selecting a preferred audio/video codec by manipulating the SDP offer packet:

https://github.com/microsoft/MixedReality-WebRTC/blob/7c588133565021e703cd1c24439b91c7a6aad523/libs/Microsoft.MixedReality.WebRTC/PeerConnection.cs#L1177-L1186

This is also exposed as a property in the LocalVideoSource and LocalAudioSource components of the Unity integration.

However I didn't verify yet if that guarantees hardware encoding/decoding, nor what kind of performance and thermal gain we get from this.

djee-ms commented 5 years ago

Update: Hardware encoding is confirmed to work on HoloLens 2 simply by using PreferredVideoCodec = "H264", without any other change.

jahnotto commented 4 years ago

Update: Hardware encoding is confirmed to work on HoloLens 2 simply by using PreferredVideoCodec = "H264", without any other change.

Decoding as well?

djee-ms commented 4 years ago

I didn't close this issue because I am sure at 99.99% it is, but I need to run that against the Media Foundation team at some point to validate 100% this is indeed the case (I already got that validation for the encoder). But yes, the code is in place and unless there is a bug in the decoder configuration I am not aware of, the decoder is also currently configured to use hardware decoding where available.

djee-ms commented 4 years ago

Update: hardware H.264 encoding is confirmed on UWP as stated above, but hardware decoding was never confirmed, and shall be assumed not present. We have no official visibility/plans on adding it at this time, due to resourcing, so removing any milestone reference for this feature.

jahnotto commented 3 years ago

Any updates on hardware decoding of H.264 and/or H.265 on HoloLens 2? @djee-ms

djee-ms commented 3 years ago

Amazingly, the Media Foundation team has been unable to provide us with a reliable method to determine which decoder is used internally when using their API (hardware or fallback software). It seems their API acts as a complete blackbox, and even with ETW traces we can't tell anything about its functioning. Since I never heard about any decoding performance issue from any user, and never saw it appear in profiling myself, we have reasonable confidence that hardware decoding is indeed used, because we expect software H.264 decoding would show up in profiling. But we can't confirm it 100%, that's why we didn't close this issue.

If you have some performance issue decoding please provide us with info and we can investigate. Otherwise unfortunately there is little chance we are looking into a 100% confidence answer to this anytime soon if this doesn't prove an issue in practice for anyone.