Seeking should start playing immediately from closest video Key-frame, or are we missing something?

[mazzomazzo]

Chrome and Edge both have this issue: when a user seeks in a local buffered range, the playback sometimes continues from video p-frame; sometimes video freezes for 1-2 sec until a key-frame is reached.

Please take a look at these two live streams: http://umediaserver.net/umediaserver/demohtml5player.html http://umediaserver.net/umediaserver/demoiptvhtml5.html

While playing, try to seek in a local buffer (it contains 1 minute of latest content) and you will see the issue. Both of these streams have Key-frame frequency of 3-4 seconds. It seems like when the player seeks by setting video.currentTime = value, then the player picks a frame at this exact commanded value, even if it is a p-frame. So you either start playing from p-frame or wait until playback reaches key-frame.

Why the player is not jumping to the closest key-frame to start playing from there?

Thank you. UMedia team

[dwsinger]

If the buffer is local, perhaps it would be better if the player 'pre-rolled' from the closest preceeding I-frame, and then started playback from the requested time?

[jyavenard]

If the buffer is local, perhaps it would be better if the player 'pre-rolled' from the closest preceeding I-frame, and then started playback from the requested time?

This picked my curiosity.. how could you seek any differently?

When seeking, AFAIK, you always find the first keyframe preceding seek time and start decoding from there. With MSE, you're guaranteed to find in a sourcebuffer a keyframe prior keytime.

[mazzomazzo]

Yes, this is quite obvious, so I cannot explain why Chrome, IE and Edge have this issue.

JY, Firefox seeks just fine with these two demo streams. Waiting on input from Chrome and Edge people.

[mazzomazzo]

David Singer: thanks for participating in this discussion. Could you please tell us - are there any plans by Apple to support MSE on iOS? The above two streams would play absolutely everywhere except browsers on iOS. The first stream is a near-real time IP camera streaming with 0.5 sec latency. Apple does not provide any way for browser-based streaming with low latency. Supporting MSE would solve that problem. I know Apple is going to support WebRTC on iOS, but WebRTC handles different scenarios (videoChat and other browser-encoded video scenarios) . For non-browser-originated streams, please support MSE!

[dwsinger]

alas, I am formally constrained from making promises or announcements....sorry

[wolenetz]

https://github.com/w3c/media-source/issues/179#issuecomment-292065493 describes Chrome's behavior. The problem, in this particular case (from examining http://umediaserver.net/umediaserver/demohtml5player.html) is that:

• Every frame in the stream is marked as a keyframe, but the decoder disagrees (on seeking, it will start prerolling the decoder from within 0.067s prior to the seek target (again, specific to the timestamp intervals in this case), but the decoder won't actually emit anything until the next keyframe is encountered).
• Once that next keyframe's decode output is available, the timestamp for that frame is significantly in the future versus the seek target time. Chrome awaits the time delta to elapse until it renders that first post-seek successful decode frame.

Solution:

• Don't mark all your frames as keyframes, if they aren't keyframes.

If your stream were audio/video, it would be much more apparent that audio would begin playing immediately post-seek, while video would vary the post-seek playback start depending on the PTS interval between the seek target and the next actual key frame after the incorrectly marked keyframe at or preceding the seek target time.

I should note that not all media engines behave well when starting decode at a non-keyframe. It is thus important to correctly mark your keyframes in the bytestream.

(Correction: almost every frame is marked as keyframe. In a 10 second clip from that repro link, only 1 wasn't marked as a keyframe.)

[jyavenard]

FWIW, since Firefox bug 1300296 (https://bugzilla.mozilla.org/show_bug.cgi?id=1300296); we've stopped relying on the container's keyframe flag. Instead we look at the H264 bytestream and look for I-frame and only rely on this information to consider if a frame is a keyframe or not.

[mazzomazzo]

It is thus important to correctly mark your keyframes in the bytestream

Of course. And frames are marked correctly in the bytestream indeed. They are marked incorrectly at the mp4 container's flag, and apparently Chrome and Edge rely on the container flag. So, of course, it seems to us that Firefox's approach is the correct one, but we will not argue much. It is an easy fix for us to provide the correct value in the container, and that's what we will do.

[dwsinger]

Firefox is recovering from a bad file, but indeed, don't make bad files in the first place...