search

w3c / media-source

Media Source Extensions
https://w3c.github.io/media-source/
Other
268 stars 57 forks source link

MSE with variable frame rate videos #176

Closed NatalyMagluy closed 3 years ago

NatalyMagluy commented 7 years ago

I didn't find any documentation stating if variable frame rate (VFR) is supported by HTML5 video+MSE. I see problems in VFR stream playback in Chrome and IE.

Can anyone tell me if VFR should be supported?

Thanks, Nataly.

jyavenard commented 7 years ago

define frame. There's nothing in the spec that specifies a particular frame rate or define it per say. The spec is all about samples having a start time and a duration

In Firefox, frame rate (aka sampling rate) for audio track must be constant between init segments. It won't play nice otherwise.

For video however, only the start time of the video frame matters. A video frame is displayed until a new one starts. The duration of a frame is as such nextFrame.startTime - currentFrame.startTime

NatalyMagluy commented 7 years ago

We have stream with PTS values as following: 133411486861242 to 133411486864245 = 3003 133411486864245 to 133411486868749 = 4504 133411486868749 to 133411486871752 = 3003 133411486871752 to 133411486876257 = 4505 133411486876257 to 133411486879260 = 3003 133411486879260 to 133411486883764 = 4504 133411486883764 to 133411486886767 = 3003 133411486886767 to 133411486891272 = 4505 133411486891272 to 133411486894275 = 3003 ...

And sourceBuffer.buffered shows gaps every ~ 2 secs (which is duration of segments)

wolenetz commented 7 years ago

For Chrome, please file a bug at https://crbug.com with media that reproduces the problem, and we'll investigate. Feel free to send email to me directly with a link to repro media, if sharing it widely is a concern. In general VFR video should not be a problem in Chrome. Does chrome://media-internals give any warnings or errors? Note especially that MSE is quite strict about when a gap (discontinuity in buffered ranges) might be detected in the appended media. Especially, if there is a jump forward more than 2*last frame duration in the sequence of DTS (not PTS), then the spec indicates discontinuity has occurred. This is the most likely source of gaps, and isn't necessarily rated to VFR.

jyavenard commented 7 years ago

AFAIK, Chrome and Edge have currently a bug open in that they incorrectly use the DTS to calculate the buffered range. This is probably what you're seeing See https://github.com/w3c/media-source/issues/54#issuecomment-215533484 (chrome tracking bug is https://bugs.chromium.org/p/chromium/issues/list?q=label:MSEptsdtsCleanup)

Firefox with those timestamps, shouldn't show gaps in the buffered range. Regardless, this wouldn't affect playback.

wolenetz commented 7 years ago

@ https://github.com/w3c/media-source/issues/176#issuecomment-268911473 : You're correct about the Chrome bugs (I'm working on those now :)). However, the MSE spec describes the discontinuity detection within a sequence of appended frames to be based upon DTS. The other half of the problem is overlap-detection in the already-buffered media, which is known in Chrome to be based incorrectly on DTS at the moment as noted in those bugs.

jdsmith3000 commented 7 years ago

Did a crbug get entered on this? I'm not sure of IE behavior, but we expect that Edge would play variable frame rate with no special limitations.  I'd be interested in getting content if anyone observes otherwise.

NatalyMagluy commented 7 years ago

@jdsmith3000 No bugs got entered yet, we're investigating how our PTS/DTS are populated to eliminate encoding issue. It's currently a Widevine only stream, so couldn't check in Edge.

wolenetz commented 6 years ago

@NatalyMagluy Please retry with Chrome Canary or Dev (M-66) with command-line option "--enable-features=MseBufferByPts". That switch engages new logic for compliant buffering by PTS intervals (not DTS intervals) in Chrome MSE. We'll be starting experiments in preparation for launch of that logic soon, too.

wolenetz commented 3 years ago

Please re-open this issue if there is still some concern around lack of variable frame rate support. The previously referenced MSE implementation issues around PTS/DTS have been fixed in Chromium for well over a year now.