Open youennf opened 1 year ago
AFAICT, there is no interaction. WebCodecs has the isConfigSupported
API, which returns whether the configuration is supported, as well as the configuration set by the Audio/Video Encoder/Decoder (which may be a subset of the config passed as an argument). So isConfigSupported
provides the information that Media Capabilities would return (other than smooth) and adds information that MC would not provide (the set configuration).
isConfigSupported
provides the information that Media Capabilities would return (other than smooth)
And powerEfficient.
Discussed at TPAC, we might want to reword the 'webrtc' category, or mention that it can be also used for live transmission via other APIs like WebCodecs.
We can amend the spec to state that this entry also works for RTC use-cases using WebCodecs. Adding new entry for "webcodecs" seems problematic because WebCodecs gives a lot of control to the website and it's not clear what it would mean in this case.
@Djuffin The answers Media Capabilities provides for 'webrtc' won't necessarily apply to WebCodecs. For example, WebRTC supports spatial scalability modes, but WebCodecs does not.
@youennf isConfigSupported
takes the configuration as an argument. So if you set hwAcceleration
= "prefer-hardware" and the configuration was supported, wouldn't that indicate that the configuration was powerEfficient
?
For example, WebRTC supports spatial scalability modes, but WebCodecs does not.
isConfigSupported
is the way to figure out if anything is supported.
Media capabilities can be used to check if it's going to be smooth for RTC scenarios.
If Media Capabilities indicates that a config will be smooth for WebRTC, does that necessarily imply that it will be smooth for WebCodecs?
What would "smooth" mean in relation to WebCodecs, where the rendering of the content is controlled by the application?
The way I meant "smooth" for WebCodecs (here https://github.com/w3c/media-capabilities/issues/202#issuecomment-1714116249) was
Maybe we should avoid dragging webcodecs into media capabilities. (Although web developers most likely will use webrtc as a proxy for webcodecs performance anyway)
We could also amend VideoEncoder.isConfigSupported and VideoDecoder.isConfigSupported to clarify that if latencyMode=realtime
or optimizeForLatency=true
it should only return true
if the device is capable of encoding/decoding in real time at a given FPS, the only difficulty here is that UA might not have a practical and reliable way to know this (but the same is true for media capabilities as well).
isConfigSupported
provides additional functionality beyond what MediaCapabilities provides (config
). Since WebCodecs decoder can be used with multiple rendering mechanisms (e.g. canvas, mediacapture-transform, WebGPU), the definition of "smooth" is not as clear as it is with a higher-level API like WebRTC.
Discussed today and it seems we could do the following:
Also, we could:
isConfigSupported
. For example, with respect to framerate
in RTC scenarios, don't return supported
if the encode/decode config indicates a desire for realtime latency but the desired framerate is unattainable.
- For example, with respect to
framerate
in RTC scenarios, don't returnsupported
if the encode/decode config indicates a desire for realtime latency but the desired framerate is unattainable.
In that case, I would think MC would return not supported for webrtc
as well.
In case MC returns smooth=false, I would think it means that the webcodecs corresponding encoder is likely to drop frames but not systematically.
Minutes from yesterday's (9 Jan 2024) discussion: https://www.w3.org/2024/01/09-mediawg-minutes.html
https://www.w3.org/TR/media-capabilities/#dom-mediaencodingtype-webrtc states that:
webrtc is used to represent a configuration that is meant to be transmitted using [RTCPeerConnection](https://www.w3.org/TR/webrtc/#dom-rtcpeerconnection) as defined in [[webrtc]](https://www.w3.org/TR/media-capabilities/#biblio-webrtc))
It does not seem that webrtc encoding mode should be used for WebCodecs+WebTransport. Is there a need to clarify this mode can also be used for other pipelines like WebCodecs+WebTransport? Or is there some benefit to introduce a specific WebCodecs mode?