What does sender.generateKeyFrame() do when sender.track is null, ended or disabled?

w3c / webrtc-encoded-transform

WebRTC Encoded Transform

https://w3c.github.io/webrtc-encoded-transform/

Other

123 stars 27 forks source link

What does sender.generateKeyFrame() do when sender.track is null, ended or disabled? #185

Closed jan-ivar closed 1 year ago

jan-ivar commented 1 year ago

Is sender.generateKeyFrame() expected to work if we have a null send track?

Maybe yes since it says it "MUST send black frames (video)" and "SHOULD send one black frame per second"?

fippo commented 1 year ago

if the encoder has no input it can not produce a frame? Should I mention that awaiting generateKeyFrame is a flawed idea?

jan-ivar commented 1 year ago

The input would be black frames no?

youennf commented 1 year ago

The input would be black frames no?

Only for muted/enabled=false track. For null tracks, If [track](https://w3c.github.io/webrtc-pc/#dom-rtcrtpsender-track) is null then the [RTCRtpSender](https://w3c.github.io/webrtc-pc/#dom-rtcrtpsender) does not send.

This paragraph seems to induce that black frames are sent for ended tracks. I am not sure browsers are doing this though (safari does not AFAIK except maybe if the track is muted/disabled while ended), we should check this.

My understanding matches @fippo's understanding, the promise will resolve when a key frame is generated, which may be very long if the track is not outputting any frame (canvas capture say).

For a null track, we can do whatever we want, the current behaviour will wait for a frame, which can only happen if an actual track is set via replaceTrack.

jan-ivar commented 1 year ago

Thanks @youennf for correcting that null should not send. I misread that. I suppose step 1 is figuring out what the different browsers emit for disabled, ended, and null. But since all but ended may be transient, the answer may be that the promise simply doesn't resolve for a long time or ever.

jan-ivar commented 1 year ago

Actually even ended is transient here if replaceTrack is used.

youennf commented 1 year ago

Are we good there or do we need any clarification in the spec?

jan-ivar commented 1 year ago

Promise will not resolve and we can close this issue.

youennf commented 1 year ago

Summary:

If the track is disabled and sender is sending, black frames will be generated and generateKeyFrame will trigger a key frame and will resolve
If the sender is not sending any frame, generateKeyFrame will never resolve.

alvestrand commented 1 year ago

Note: in other contexts, we (Chrome) have found it necessary to generate a frame when a keyframe is asked for, even if we would normally not do so (for instance when capturing a tab that doesn't have any change).