Clarification needed for HTMLVideoElements that are playing a MediaStream

[Pehrsons]

If a video element is playing a MediaStream containing a live video MediaStreamTrack, how should that affect the dropped video frame count and the corrupted video frame count?

These tracks are not decoded by the video element itself, so how to handle drops/corruptions by the decoder is not immediately clear.

[Joe-Palmer]

I would say the most important stat for live video is the total frame count (which is currently only accessible with mozPaintedFrames in Firefox and webkitDecodedFrameCount in Chrome). If the dropped and corrupted frame counts do not make sense in this case, I don't see an issue in having them report 0 (as they do now).

[Pehrsons]

With corruptVideoFrames deprecated, this relies less on the assumption that a decoder is present.

It does still say It is incremented when a video frame is dropped predecode. in the dropped video frame count rules, which assumes a decoder is present for the video element. This spec should define "predecode" to clarify this.

To highlight the issue here, a MediaStream with a video MediaStreamTrack originating from an RTCPeerConnection uses a decoder. Should frames dropped by that decoder increment the counter? How far does the pre in predecode stretch? FWIW, an RTCPeerConnection has its own stats API for understanding how data is dropped by the sender and on the wire.

The media element spec distinguishes local from remote media resources which could be a handy definition to use, but then MediaSource (MSE) is just like MediaStream a local media resource, and I assume the dropped frame count should apply there.

@chcunningham thoughts?

[chcunningham]

Hi, apologies for the wait. Busy w/ paternity leave and work on other specs.

It does still say It is incremented when a video frame is dropped predecode. in the dropped video frame count rules, which assumes a decoder is present for the video element. This spec should define "predecode" to clarify this.

I support adding some clarification. The origins of MPQ are MSE, so I that language probably wasn't considering usage w/ MediaStreamTrack. My guess is that mentioning "predecode" was to cover cases where UAs respond to decoder slowness by dropping frames without even trying to decode (skipping to the next iframe). I think Firefox does it this way.

Given the ambiguity, UAs may have implemented dropped frame counts for MediaStream

If sites aren't much using this for MediaStream sources, I'd vote to clarify that the counts should be zero here and should instead come from RTCPeerConnection. My second choice would be to clarify that the numbers are the same as from RTCPeerConnection.

[Pehrsons]

FWIW returning numbers here that stem from RTCPeerConnection brings its own problems. To exemplify: what if the peer connection has run for an hour before the track starts playing in the media element?

[beaufortfrancois]

@eladalon1983 Would you be able to help with that by any chance or know who could?

[jan-ivar]

I'd vote to clarify that the counts should be zero here and should instead come from RTCPeerConnection.

This seems like the simplest answer. A MediaStreamTrack can come from a wealth of sources, so it's going to be hard to specify this for all of them.

*Missing from this diagram is VideoTrackGenerator

[eladalon1983]

@eladalon1983 Would you be able to help with that by any chance or know who could?

I'd first check with @henbos for stats.

[jan-ivar]

I'd vote to clarify that the counts should be zero here and should instead come from RTCPeerConnection. My second choice would be to clarify that the numbers are the same as from RTCPeerConnection.

what if the peer connection has run for an hour before the track starts playing in the media element?

The third option: numbers start counting at different times, but increment at the same rate during playback (e.g. off by one hour's worth in the above example).

This time horizon difference is observable with gUM and track.stats already.¹ ² E.g.

const stream = await navigator.mediaDevices.getUserMedia({video: true});
const [track] = stream.getVideoTracks();
await wait(2000);
video.srcObject = stream;
await wait(2000);
dump(() => video.getVideoPlaybackQuality().totalVideoFrames);
dump(() => track.stats);

...produces in Chrome:

video.getVideoPlaybackQuality().totalVideoFrames = 47
track.stats.deliveredFrames = 93

But comment out the first await wait(2000); and it produces:

video.getVideoPlaybackQuality().totalVideoFrames = 46
track.stats.deliveredFrames = 46

This makes sense to me as it answers slightly different questions.

---

_{1. the fiddle shows totalVideoFrames not droppedVideoFrames since that would have taken a long time to repro, but I think the principle remains the same: only count drops happening in the relevant time window 2. track.stats are currently only defined for tracks from getUserMedia and getDisplayMedia. i.e. camera, microphone and screen-sharing tracks}