search

WebPlatformForEmbedded / WPEWebKit

WPE WebKit port (downstream)
213 stars 136 forks source link

[wpe-2.38] media_content_types_requiring_hardware_support and audio decoders #1122

Closed jakub-gocol-red closed 1 year ago

jakub-gocol-red commented 1 year ago

WPE-2.38 has webkit_settings_set_media_content_types_requiring_hardware_support function which allows to set pattern that is matched against codecs provided in MediaSource.isTypeSupported api. If there's a match, such codec will only by marked as supported, if there is hardware decoder available for it.

In rdk based projects such as mine, pattern is set here to "video/*". I guess the idea was to require hardware decoder for all video codecs.

But it's very common for applications to call MediaSource.isTypeSupported with 'video/mp4; codecs="avc1.42E01E,mp4a.40.2"' as parameter. This matches "video/*" pattern for both codecs so both of them are required to have hardware decoder. But in here, where browser precomputes what codecs are supported and which of them have hardware decoders, it's just assumed that audio codecs don't have hardware decoders and MediaSource.isTypeSupported('video/mp4; codecs="avc1.42E01E,mp4a.40.2"') returns false.

  1. Requiring hardware decoders for all video codecs and nothing else seem like the most common use case. With current implementation setting pattern to "video/*" is not working as intended. Is there any recommended pattern that we should use?
  2. Should this pattern even be checked against audio codecs? Based on this code, noone expects them to have hardware decoding.
eocanha commented 1 year ago

This feature was added when there was still no support for SourceBuffers with more than one stream (and therefore multiple codecs). Let me think of a way to make it compatible with multiple streams and I'll comment about it.

eocanha commented 1 year ago

I've been experimenting a bit with how the media_content_types_requiring_hardware_support evaluation works and I've come up with a string that works in your specific case. The corresponding WPEFramework/plugins/WebKitBrowser.json line should remain like this:

    "mediacontenttypesrequiringhardwaresupport": "video/*; codecs=\"avc*\":video/*; codecs=\"vp*\":video/*; codecs=\"av1*\"",

You should add any other codec that your hardware may support.

This setting gives the following result on my Raspberry Pi 3 here, which only supports h264 on hardware. I think this matches with your expectations:

jakub-gocol-red commented 1 year ago

thanks, it's a bit tedious process, but it indeed will work.