Controlling channel mechanism is unclear

[youennf]

The spec is not clear about the controlling channel mechanism, in particular:

1. its impact on existing specs. Should specs define when native sinks create such signals? Should specs define what native
2. its impact on users. What happens if users do not handle them? Is it future proof? What will users be able to do if using them?

Also internal signals have the potential for UA specific behaviours so it is unclear whether that is a good idea to expose this. One possibility would be to identify and document use cases before documenting the API shape.

[guidou]

The purpose of the signaling mechanism is to expose communication from sinks to sources that may be occurring. For example, in Chromium, a Peer Connection sometimes internally requests a frame from the source as part of its internal implementation. It also sends other feedback to capturers. Some sources ignore this feedback and others act on it. An implementation is not required to do these things, but it can do them in order to, for example, optimize performance.

What the signaling mechanism allows is to expose this to custom sources and sinks. A source (e.g., the writable of a MediaStreamTrackGenerator ) can ignore a request for a frame if it doesn't make sense for it.

So, to answer your questions.

1. No. Specs shouldn't define when native sinks need to create such signals. A native implementation of a sink may send them if it makes sense, but it is not required to.
2. Signals can be ignored since not all sources (platform or JS) are required to act on them.

[youennf]

1. A native implementation of a sink may send them if it makes sense, but it is not required to.

I fear this is a recipe for interop issues and bad portability of the same web page/various browsers or various platforms. Some web pages may for instance expect some signals that will not be provided by some UAs/platforms.

For instance, it is not clear when web pages should expect "request-frame" or when implementations should send this signal. If "request-frame" can safely be ignored, why introducing it?

That does not mean it is a bad idea to expose such signals. To make progress there, I think we should try to first more precisely define sink and source models than what mediacapture-main is defining. This is not a simple exercise though since browsers might have different understanding and implementations. By doing this exercise, we should be able to get consistent support across browsers.

[guidou]

1. A native implementation of a sink may send them if it makes sense, but it is not required to.

I fear this is a recipe for interop issues and bad portability of the same web page/various browsers or various platforms. Some web pages may for instance expect some signals that will not be provided by some UAs/platforms.

For instance, it is not clear when web pages should expect "request-frame" or when implementations should send this signal. If "request-frame" can safely be ignored, why introducing it?

Safely ignored means in this context that correctness would not be affected by ignoring a signal. Handling the signal might result in better performance by the sink.

That does not mean it is a bad idea to expose such signals. To make progress there, I think we should try to first more precisely define sink and source models than what mediacapture-main is defining. This is not a simple exercise though since browsers might have different understanding and implementations. By doing this exercise, we should be able to get consistent support across browsers.

I agree that having a more precise definition of sinks and sources would be a good idea in general and would specifically help define a signaling mechanism.

[youennf]

Handling the signal might result in better performance by the sink.

Understood, though if web page implements audio glitch-free rendering based on using that signal on a specific browser, it becomes important for UAs to support this mechanism using the same algorithms/timing. Ditto if we are talking about preserving battery life in case, say rendering can increase/decrease its frame rate.

As an example, some websites have good quality calls (say ramp-up) if UAs have various tools like TWCC or REMB. This makes having these tools part of 'correct' user experience, even though this is not a correctness issue per se.

[guidou]

Handling the signal might result in better performance by the sink.

Understood, though if web page implements audio glitch-free rendering based on using that signal on a specific browser, it becomes important for UAs to support this mechanism using the same algorithms/timing. Ditto if we are talking about preserving battery life in case, say rendering can increase/decrease its frame rate.

As an example, some websites have good quality calls (say ramp-up) if UAs have various tools like TWCC or REMB. This makes having these tools part of 'correct' user experience, even though this is not a correctness issue per se.

I don't disagree with what you say here, but my impression is that the MediaStream-related specs generally do not define specific performance requirements as correctness, so I would see the glitch/battery examples you mention as a quality issue more than a correctness issue.

[guidou]

Closing this issue, since we have removed the controlling channel mechanism.