Different behavior when onaddtrack is fired for element.captureStream

bc-lee commented 1 month ago

Consider the following code:

<!DOCTYPE html>
<html>
<head>
    <title>captureStream for Video Element</title>
</head>
<body>

<video id="video"></video>
<button onclick="streamFromVideo()">Stream from Video Element</button>

<script>
  function streamFromVideo() {
    const video = document.getElementById('video');
    video.src = './test-av-384k-44100Hz-1ch-320x240-30fps-10kfr.webm';
    video.onerror = (error) => {
      console.error('Error:', error);
    };
    if (typeof video.captureStream !== 'function' && typeof video.mozCaptureStream === 'function') {
      HTMLVideoElement.prototype.captureStream = HTMLVideoElement.prototype.mozCaptureStream;
    }

    const stream = video.captureStream();
    stream.onaddtrack = () => {
      console.log('onAddTrack');
      const tracks = stream.getTracks();
      for (const track of tracks) {
        console.log('Track readyState:', track.readyState);
        console.log('Track kind:', track.kind);
      }
      console.log('Video tracks:', stream.getVideoTracks().length);
      console.log('Audio tracks:', stream.getAudioTracks().length);
    };
  }
</script>
</body>

The test video file test-av-384k-44100Hz-1ch-320x240-30fps-10kfr.webm has both audio and video tracks. This code is almost identical to the one in the web platform test.

Below are the logs from the code above:

Chrome 125.0.6422.141:

onAddTrack
Track readyState: live
Track kind: audio
Track readyState: live
Track kind: video
Video tracks: 1
Audio tracks: 1
onAddTrack
Track readyState: live
Track kind: audio
Track readyState: live
Track kind: video
Video tracks: 1
Audio tracks: 1

Firefox 126.0:

onAddTrack
Track readyState: live
Track kind: audio
Video tracks: 0
Audio tracks: 1
onAddTrack
Track readyState: live
Track kind: audio
Track readyState: live
Track kind: video
Video tracks: 1
Audio tracks: 1

In Chrome, the onaddtrack is fired twice, however even with the first event, both audio and video tracks are present in the stream. In Firefox, the onaddtrack is fired twice, but the first event only has the audio track, and the second event has both audio and video tracks.

I wonder which behavior is correct. The current web platform test assumes that the behavior in Chrome is correct (it checks only the first event), but it may be an implementation detail of Chrome.

steely-glint commented 1 month ago

I think you should probably look at the event. The code above is looking at the state of the stream.

stream.onaddtrack = (ev) => { console.log('onAddTrack kind: '+ev.track.kind );

Pehrsons commented 1 month ago

I would agree the spec can be clearer on this.

I think the question boils down to whether the events fire sync or async in the text:

If the selected VideoTrack or enabled AudioTracks for the media element change, a addtrack event with a new MediaStreamTrack is generated for each track that was not previously selected or enabled; and a removetrack events is generated for each track that ceases to be selected or enabled. A MediaStreamTrack MUST end prior to being removed from the MediaStream.

I'll also note that the addtrack event on AudioTrackList and VideoTrackList in the media element spec is fired sync, while the change event on the same objects is fired async.

w3c / mediacapture-fromelement

Different behavior when onaddtrack is fired for element.captureStream #100