search

w3c / webrtc-pc

WebRTC 1.0 API
https://w3c.github.io/webrtc-pc/
Other
433 stars 115 forks source link

Proposing setCodecPreferences to deal with both send and recv codecs #2939

Open henbos opened 4 months ago

henbos commented 4 months ago

Based on prior discussions such as https://github.com/w3c/webrtc-pc/issues/2937#issuecomment-1942031155, here's a concrete proposal which I think fixes unidirectionality, requires no changes to JSEP (consistent with https://github.com/w3c/webrtc-pc/issues/2938#issuecomment-1942116384) and avoids backwards compat issues introduced by #2926.

How did we get here?

JSEP has this note (https://rtcweb-wg.github.io/jsep/#rfc.section.4.2.6):

Note that setCodecPreferences does not directly affect which codec the implementation decides to send. It only affects which codecs the implementation indicates that it prefers to receive, via the offer or answer.

We took this to mean "SCP should only take receive codecs as input", something which JSEP neither says or IMO imply. The note only talks about the direct effect; there's also the indirect effect that the preferences have.

Excluding send codecs was a mistake.

The preferences are relevant even for sendonly transceivers, since we must have something to offer (https://rtcweb-wg.github.io/jsep/#rfc.section.5.2.1):

If codec preferences have been set for the associated transceiver, media formats MUST be generated in the corresponding order, and MUST exclude any codecs not present in the codec preferences.

This does not mean that preferences are irrelevant for the sending use case: the receiver not having called setCodecPreferences is a valid use case, in which case the sender's preferences are the ones we use (https://rtcweb-wg.github.io/jsep/#rfc.section.5.3.1):

  • If codec preferences have been set for the associated transceiver, media formats MUST be generated in the corresponding order, regardless of what was offered, and MUST exclude any codecs not present in the codec preferences.
  • Otherwise, the media formats on the m= line MUST be generated in the same order as those offered in the current remote description, excluding any currently unsupported formats. Any currently available media formats that are not present in the current remote description MUST be added after all existing formats.

We could simply say we don't care about this use case, but that appears to have some downsides when it comes to unidirectional codecs.

Proposal

"setCodecPreferences accepts both send codecs and receive codecs as input."

The actual preferences you end up using depend on the createOffer filtering we already apply based on transceiver.direction. This makes it valid for...

To be discussed

Note that it's possible for the filtering to end up with an empty codec list. There are multiple ways we could deal with this, of which I have no strong preference. Examples:

  1. SCP and direction setting could sanity check and throw if we end up with an empty set.
  2. createOffer could throw.
  3. We don't throw, but if we have nothing to offer we reject the m= section.
  4. If we end up with empty set we offer one of the mandated to support sendrecv codecs as a "backup" (VP8 and H264).
  5. SCP could require that at least one codec is sendrecv, in which case there is no filtering that could remove it, and changing direction is always safe.
fippo commented 4 months ago

we must have something to offer

We don't. If we have nothing to offer this is similar to "we have nothing to answer" which is described as

None of the offered media formats are supported and, if applicable, allowed by codec preferences.

in https://rtcweb-wg.github.io/jsep/#rfc.section.5.3.1 I think this case is rare enough that going for 3 is a pragmatic solution. And yes, I have a code change for that already that is quite small

aboba commented 4 months ago

@henbos You may be taking JSEP Section 4.2.6 too literally. It is correct in saying that sCP is not for selecting what is sent. That is true regardless of direction. However, saying that sCP only applies to codecs that can be received, regardless of direction, has to contend with RFC 3264 Section 5.1:

"For a sendonly stream, the offer SHOULD indicate those formats the offerer is willing to send for this stream. For a recvonly stream, the offer SHOULD indicate those formats the offerer is willing to receive for this stream. For a sendrecv stream, the offer SHOULD indicate those codecs that the offerer is willing to send and receive with."

Since this is a SHOULD, there is some wiggle room for sendrecv streams. For example, a sendrecv m-line with preference {H.265, H.264} could be allowed if H.265 can be received but not sent. This would allow negotiation to succeed between a receive-only H.265 Offerer and an Answerer who can send and receive both H.265 and H.264, using only a single sendrecv m-line, as long as the Answerer included both H.265 and H.264 in its Answer. Since non-WebRTC endpoints may not support sendonly and recvonly m-lines, this approach has interoperability advantages.

However, concluding that sCP with a sendonly m-line is only about receiving preferences seems like a step too far. Instead I'd suggest that a sendonly transceiver can only offer codecs/profiles that can be sent (send-only or send/recv). In a situation where the browser can send at a different profile/levels than it can receive, only the send profiles can be included when direction = sendonly.

Similarly, a recvonly transceiver can only offer codecs/profiles that can be received (recv-only or send/recv). In a situation where the browser can send at a different profile/levels than it can receive, only the receive profiles are included. This is consistent with JSEP Section 4.2.6.

dontcallmedom-bot commented 4 months ago

This issue was discussed in WebRTC February 2024 meeting – 20 February 2024 (setCodecPreferences to deal with both send and recv codecs #2939)

jan-ivar commented 3 months ago

@henbos what are next steps here?

aboba commented 3 months ago

Related: https://github.com/aboba/hevc-webrtc/issues/22

Justin's suggestion.

aboba commented 2 months ago

@jan-ivar H.265 is getting close to being enabled behind a flag, so in the near future we will be able to see how receive-only codecs behave, and what the problems are. At that point we can present the issue to the WG in light of the behavior we see. IMHO it would be desirable to converge on recommended changes prior to enabling H.265 by default.