search

video-dev / hls.js

HLS.js is a JavaScript library that plays HLS in browsers with support for MSE.
https://hlsjs.video-dev.org/demo
Other
14.8k stars 2.57k forks source link

Seeking to end of previous fragment results in media error, or wrong frame being displayed #2327

Closed MsLizzie closed 3 years ago

MsLizzie commented 5 years ago

What version of Hls.js are you using?

The latest version: https://cdn.jsdelivr.net/npm/hls.js@latest

What browser and OS are you using?

Safari / Chrome / Firefox, Windows 10 / Mac High Sierra. Exact symptoms are browser specific.

Test stream:

https://d25uzy1v1dityp.cloudfront.net/master_playlist.m3u8

Test page, demonstrating issue

https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html

Checklist

General description of issue

When seeking from fragment X to a point near the end of fragment X - 1, the correct frame is not always displayed. The actual result is browser specific, and not 100% consistent. My test page (https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html) can be used to demonstrate the problem easily.

The test page (https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html)

The test page contains 3 buttons, allowing seeking to 3 specific frames in the video. Each frame of the video has burned-in time code to make it easy to check which frame is being displayed. If the page is reloaded before pressing each button, then the correct frame is seeked to in ALL browsers. To ensure that this works correctly, 2 changes to the configuration were required. 1) maxFragLookUpTolerance is set to 0.0. With the default of 0.25, any seek to a point in the last 0.25 seconds of a fragment, causes the seek to go to the first frame of the next fragment, which is not desirable. 2) nudgeOffset is set to 0.0, and nudgeMaxRetry is set to zero. This is done to prevent the wrong frame being selected in the case of a media error that is recoverable.

The problem only manifests itself when the seek buttons are pressed in sequence (left-to-right) without refreshing the page. The first seek button seeks to 3 frames before the end of fragment 231 The second seek button seeks to 3 frames before the end of fragment 230 The thrid seek button seeks to 3 frames before the end of fragment 229

Steps to reproduce

Use my test page to reproduce the issue (https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html) Press each of the seek buttons in sequence (left-to-right). Several results are possible (different results seems dependent on the amount of time you wait before pressing the 2nd and 3rd seek buttons). Refresh the page running each test below.

Firefox

Test 1) Press "Seek 55676" (which will correctly display frame 55676) then wait (say) 8 seconds, then press "Seek 55436". This results in a media error (indicated in the browser console), although frame 55436 is displayed correctly once hls.recoverMediaError() completes. Test 2) Press "Seek 55676". Once frame 55676 is displayed, immediately press "Seek 55436". This time there is no media error, however the wrong frame is displayed (55440) Test 3) Press "Seek 55676", wait 8 seconds, press "Seek 55436" wait 8 seconds, press "Seek 55196". There is no media error, but the wrong frame (552200) is displayed.

The behaviour is identical on PC (Windows 10) and Mac (High Sierra)

Chrome

Test 1) Press "Seek 55676" (displays correct frame 55676), wait 8 seconds, press "Seek 55436". This results in a media error. Frame 55436 is displayed correctly once hls.recoverMediaError() completes. Test 2) Press "Seek 55676". Once frame 55676 is displayed, immediately press "Seek 55436". After about 2-3 seconds, there will be a media error, however hls.recoverMediaError() doesn't manage to recover, and the video is completely white. There are no additional errors in the console. At this point, the video is no longer playable, and the page needs to be reloaded. Note that the behaviour described in this test occurs roughly 80% of the time on a Mac, and about 20% on a PC. The rest of the time, the video recovers, and the correct frame gets displayed.

This behaviour is reproducable on PC and Mac.

Safari (Mac)

Test 1) Press "Seek 55676". Displays frame 6 briefly, before correctly displaying 553676. This is just noted in case it has any relevance to the main bug I am submitting here. Test 2) Press "Seek 55676" then press Seek 55436. This results in a media error irresepective of the delay between pressing the buttons. Frame 55436 is displayed correctly when recoverMediaError() completes. Test 3) Press "Seek 55676", then "Seek 55436", then "Seek 55196". You will get a media error on both of the last 2 seeks. This seems 100% consistent on Safari, and recoverMediaError() always seems to work.

Expected behavior

When seeking to a specific location, the frame displayed should be consistent, and not dependent on previous seeks in the video. All browsers should behave in an identical way, and the seek should not result in a fatal media error (Hls.ErrorTypes.MEDIA_ERROR)

Actual behavior

Variable. Described in detail in the steps to reproduce, above. Either a media error occurs (which may/may not be recoverable), or the wrong frame might be displayed (e.g. in Firefox)

Other comments

This problem appears to relate to caching. In the case where you seek to a point in the previous fragment where the previous segment hasn't been cached yet, then the fragment doesn't always get downloaded, resulting in a media error. In the case that the section of video has already been played through (and therefore cached), then the seeking works perfectly in all browsers.

Media errors in Chrome

When a media error occurs in Chrome, 2 entries are added in media-internals Example below:

Entry 1

Player 650:96 (WEBMEDIAPLAYER_DESTROYED) Player properties audio_buffering_state BUFFERING_HAVE_NOTHING event WEBMEDIAPLAYER_DESTROYED pipeline_state kStopped player_id 77 render_id 650 seek_target 2309.84375 video_buffering_state BUFFERING_HAVE_NOTHING

Log 00:00:00.000 seek_target 2309.84375 00:00:00.000 pipeline_state kSeeking 00:00:00.000 audio_buffering_state BUFFERING_HAVE_NOTHING 00:00:00.000 video_buffering_state BUFFERING_HAVE_NOTHING 00:00:00.001 event WEBMEDIAPLAYER_DESTROYED 00:00:00.001 pipeline_state kStopping 00:00:00.002 pipeline_state

Entry 2

Player properties duration 2538.499999 event SUSPENDED for_suspended_start false found_video_stream true frame_title frame_url https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html height 288 info Selected MojoVideoDecoder for video decoding, config: codec: h264, format: PIXEL_FORMAT_I420, profile: h264 baseline, coded size: [512,288], visible rect: [0,0,512,288], natural size: [512,288], has extra data: false, encryption scheme: Unencrypted, rotation: 0°, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED} is_platform_video_decoder true origin_url https://d25uzy1v1dityp.cloudfront.net/ pipeline_buffering_state BUFFERING_HAVE_ENOUGH pipeline_state kSuspended player_id 87 render_id 650 seek_target 2309.84375 surface_layer_mode kAlways url blob:https://d25uzy1v1dityp.cloudfront.net/7ced3796-9c2f-414e-9511-ac7b925daee5 video_buffering_state BUFFERING_HAVE_ENOUGH video_codec_name h264 video_dds false video_decoder MojoVideoDecoder width 512

Log 00:00:00.000 origin_url https://d25uzy1v1dityp.cloudfront.net/ 00:00:00.000 frame_url https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html 00:00:00.000 frame_title 00:00:00.000 surface_layer_mode kAlways 00:00:00.000 url blob:https://d25uzy1v1dityp.cloudfront.net/7ced3796-9c2f-414e-9511-ac7b925daee5 00:00:00.000 info ChunkDemuxer: buffering by PTS 00:00:00.000 pipeline_state kStarting 00:00:00.036 found_video_stream true 00:00:00.036 video_codec_name h264 00:00:00.038 video_dds false 00:00:00.038 video_decoder MojoVideoDecoder 00:00:00.038 is_platform_video_decoder true 00:00:00.038 info Selected MojoVideoDecoder for video decoding, config: codec: h264, format: PIXEL_FORMAT_I420, profile: h264 baseline, coded size: [512,288], visible rect: [0,0,512,288], natural size: [512,288], has extra data: false, encryption scheme: Unencrypted, rotation: 0°, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED} 00:00:00.038 pipeline_state kPlaying 00:00:00.041 seek_target 2309.84375 00:00:00.041 pipeline_state kSeeking 00:00:00.041 pipeline_state kPlaying 00:00:00.081 height 288 00:00:00.081 width 512 00:00:00.085 video_buffering_state BUFFERING_HAVE_ENOUGH 00:00:00.085 for_suspended_start false 00:00:00.085 pipeline_buffering_state BUFFERING_HAVE_ENOUGH 00:00:00.392 duration 2538.499999 00:00:16.131 pipeline_state kSuspending 00:00:16.131 pipeline_state kSuspended 00:00:16.132 event SUSPENDED

MsLizzie commented 5 years ago

Do you need any further information about this issue?

itsjamie commented 5 years ago

I think there is enough information here. Thank you for the great report and test page!

robwalch commented 4 years ago

Hi @MsLizzie,

This appears to be fixed in the latest release. Can you please confirm?

https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html

If you're able to reproduce in latest and I missed something I'll reopen the issue. Thank you!

MsLizzie commented 4 years ago

Hi Rob,

Unfortunately, this issue is not resolved. It's slightly different than it was (the media errors are no longer reported), but the main issue remains. The previous behaviour where the results changed depending on how long you waited between seeks, also seems to no longer occur.

I've checked everything again, using the latest versions of browsers with the same test page: https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html

The test page uses the latest version: https://cdn.jsdelivr.net/npm/hls.js@latest

My browser versions: Chrome: 84.0.4147.89 [PC & MAC] Firefox: 78.0.2 (64-bit) [PC & MAC] Safari (macOS Mohave): Version 13.1.2 (14609.3.5.1.5) Safari (macOS High Sierra): Version 13.1.2 (13609.3.5.1.5)

I've checked using 2 Macs, and 1 PC as follows:

Chrome (Mac OS High Sierra) Chrome (Mac OS Mojave) Chrome (PC - Windows 10) Firefox (Mac OS High Sierra) Firefox (Mac OS Mojave) Firefox (PC - Windows 10) Safari (Mac OS High Sierra) Safari (Mac OS Mojave)

Note that as before, the issue seems to relate to seeking near the end of a segment when the video has not been cached yet.

On all browsers checked, if you refresh the page, then press the buttons in this order: Button 3 -> Button 2 -> Button 1, then the results are perfect, and the correct frame is seeked to on all browsers.

However, the following does NOT work:

Test: Refresh page. Click button 1 (Seek 55676). After the seek completes click button 2 (Seek 55436)

Results: Chrome (Mac OS High Sierra) - Seek to 55676 works. Does not seek at all when button 2 pressed (bug 1) Chrome (Mac OS Mojave) - Seek to 55676 works. Does not seek at all when button 2 pressed (bug 1) Chrome (PC - Windows 10) - Seek to 55676 works. Does not seek at all when button 2 pressed (bug 1) Firefox (Mac OS High Sierra) - Seek to 55676 works. Seek to 55436 seeks incorrectly to frame 055440 (bug 2) Firefox (Mac OS Mojave) - Seek to 55676 works. Seek to 55436 seeks incorrectly to frame 055440 (bug 2) Firefox (PC - Windows 10) - Seek to 55676 works. Seek to 55436 seeks incorrectly to frame 055440 (bug 2) Safari (Mac OS High Sierra) - Seek to 55676 works. Does not seek at all when button 2 pressed (bug 1) Safari (Mac OS Mojave) - Seek to 55676 works. Does not seek at all when button 2 pressed (bug 1)

So as you can see above, all browsers have failed the test, either because they display the wrong frame in the 2nd seek, or nothing happens for the second seek.

Note also that the secondary minor bug also still occurs in Safari: Refresh page, and click any button. The browser briefly wrongly displays frame 6 or frame 7 (bug 3) before displaying the correct frame.

Thanks for your help

Liz

robwalch commented 4 years ago

Hi @MsLizzie,

Frame accuracy is a browser issue (sometimes an HLS packaging issue). It's not something hls.js can guarantee.

As far as why it is not displaying the correct frame, I have an explanation below. For clarity, let's list the issues as it's tricky for me to understand what the main issue is here with all the information provided. Thanks for being patient as I catchup.

Issues:

I would ask that you try the demo Timeline to troubleshoot the issue. It should help illustrate where browsers actually seek, as well as where hls.js plots each segment according to the media it parses.

The player is seeking, but hls.js is not buffering media at the point it seeks to, and because the video is paused the video does not update. hls.js "thinks" the segment ahead of the time you are seeking to is the correct segment to buffer.

Screen Shot 2020-07-27 at 12 35 09 PM

To fix this you should try reducing maxFragLookUpTolerance in the config. Setting this to 0 will ensure that the correct segment is loading. I will keep this issue open with the focus being on improving segment selection so that we don't overshoot the selected segment for loading when seeking because of maxFragLookUpTolerance.

MsLizzie commented 4 years ago

Hi Rob,

I will take a look at the demo (with timeline), Although the only case I really care about at the moment is when the video is in pause mode, and you perform a seek. The problems occur only when the seek is near the end of a segment, so I'm not sure how I can duplicate this easily in your demo, as it doesn't seem to allow you to seek to precise locations.

FYI: For us, hls-js generally works very well, and we do rely on frame accurate seeking. Before we used hls-js, the frame accurate seeking worked perfectly in all browsers on single unsegmented video files in all cases frame rates, etc. With hls.js, this is almost true, and only fails in the case where seeks are very close to the end of a segment that hasn't been cached yet. Only then, does the behaviour become browser specific, with either the frame not being displayed at all, or the wrong frame displayed, because hls-js jumps into the next segment.

This is a very exceptional case, and it was hard for me to reproduce reliably (in our software 99%+ of random seeks worked perfectly), which is why I created my example test page when I managed to find specific seek locations that would cause problems consistently.

So it seems to me that this issue is all about hls-js overshooting the required segment. You may not be able to guarantee frame accurate seeking, but I believe that will become the outcome as far as I am concerned if this issue can be fixed.

You said, though, that setting maxFragLookUpTolerance to zero will ensure that the correct segment is loaded. I do not believe this is the case. As I mentioned at the top of this support ticket, in my test page config I use:

maxFragLookUpTolerance: 0.0 nudgeOffset: 0.0, nudgeMaxRetry: 0,

So with a maxFragLookUpTolerance of zero there are still problems seeking near the end of segments. Are there any other settings that might have an impact on this?

Thanks

Liz

SnapStreamJason commented 4 years ago

We've done @MsLizzie's test (using the exact same stream and same seek points) with video.js (current released version) and gotten perfectly frame accurate seeking results in all browsers with that, so I'm not sure just dismissing that part as a browser issue is the right answer.

robwalch commented 4 years ago

Hi Liz,

The screen shot above is the result of me seeking in the dev tools exactly as your test page does without setting maxFragLookUpTolerance to 0 - I did this to confirm the bug that needs to be addressed. When I set maxFragLookUpTolerance to 0 and do the same, then sn 230 is loaded and the frame you are looking for is rendered as far as I can tell - so there is a workaround.

Outside of that is there any other issue? If the browser sets currentTime accurately AND media is buffered at the time presented, then is the wrong frame still rendered?

MsLizzie commented 4 years ago

Hi Rob,

Once the media has been buffered, the seeking works perfectly across all browsers. The issue I am having is specifically when we seek to a location that hasn't been buffered yet.

I'm struggling to see the workaround. The problem I am seeing has always occurred when the maxFragLookupTolerance is set to zero already.

It is important to seek to exactly the locations I specified to demonstrate this, which I have now done with your demo (https://hls-js.netlify.app/demo/) as follows, by performing the seeks from the javascript console:

I changed the config in your demo player to use the same as mine, although I don't think it makes any difference.

{ "maxFragLookUpTolerance": 0.0, "nudgeOffset": 0.0, "nudgeMaxRetry": 0 }

Stream: https://d25uzy1v1dityp.cloudfront.net/master_playlist.m3u8

Make sure the video is paused as quickly as possible.

So the primary issue I have is demonstrated by following these steps:

Chrome, from JS console: $('#video')[0].currentTime = 2319.84 (should seek to 55676) You can see the segments loading in the timeline, and the player seeks correctly to frame 55676 $('#video')[0].currentTime = 2309.84(should seek to 55436) Nothing appears on the timeline, no seek occurs at all.

Firefox, from JS console: $('#video')[0].currentTime = 2319.84 (should seek to 55676) You can see the segments loading in the timeline, and the player seeks correctly to frame 55676 $('#video')[0].currentTime = 2309.84(should seek to 55436) On the timeline, you can see that the wrong segment SN 231 was loaded (not SN 230), and therefore the wrong frame gets displayed.

I've tried different combinations of configs, with different values of maxFragLookUpTolerance, but I cannot get your demo to work correctly in either Chrome or Firefox in this case. Everything behaves exactly the same way as my own test.

Hope this helps!

Thanks

Liz

MsLizzie commented 4 years ago

As a follow up to the above, I want to mention the following again, in case it was lost in the above.... The issue I've been describing only occurs depending on what the previous seek was and what the new seek destination is.

In the example above, a seek to 2319.84 seconds works, but when followed by a seek to 2309.84 seconds it always fails (when the video at 2309.84 isn't cached yet)

Whereas, if the video is loaded again, and you seek straight to 2309.84 it works fine (all browsers)

One difference is that in the first (fail) case the seek is a backwards seek from 2319.84 to 2309.84, but in the second (success) case, it is a forwards seek (from 00:00:00)

I have only ever seen the failure (so far) when you seek backwards in the video, and the destination location is not cached yet.

Could this be relevant to the problem?

Thanks

Liz

robwalch commented 4 years ago

I have only ever seen the failure (so far) when you seek backwards in the video, and the destination location is not cached yet. Could this be relevant to the problem?

Yes. The issue is that the segment needed to display the frame is not buffered when the playhead only passes (close to) the end of the segment, and the next segment is buffered.

I see that with maxFragLookUpTolerance: 0, seeking forwards is fixed, but not seeking back to the end of a fragment that precedes one that has.

This is an issue in the stream controller's _doTickIdle where it thinks it has enough buffered ahead. The other problematic config option here is maxBufferHole. The current segment is not loaded when the next buffered range (next segment that was already loaded before seeking back) is less than maxBufferHole from currentTime.

So, when seeking to a position near the end of a segment:

Set maxBufferHole to 0 (along with maxFragLookUpTolerance) to workaround this issue for the time being.

MsLizzie commented 4 years ago

Hi, thanks for the information. I am still experiencing issues, however.

Has something changed in your demo player since last week? It's now behaving differently than before:

Considering again Seek 1: 2319.84 seconds followed by Seek 2: 2309.84 seconds (uncached)

Old behaviour: Chrome: Seek 1 works, seek 2 didn't seek Firefox: Seek 1 works, seek 2 seeks to wrong location

Today's new behaviour: Chrome: Seek 1 fails Firefox: Seek 1 works, seek 2 fails and displays a spinner forever over the player.

This new behaviour occurs whether you set maxBufferHole to zero or not.

However, today I am seeing a divergence between my player test page and your demo player for the first time: These 2 links go to my test page without and with maxBufferHole set to zero respectively

https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue_buffer_hole_0.html

When I set maxBufferHole to zero, there IS an improvement on my test page. Chrome, now works perfectly.

However it does NOT fix the Firefox issue, and it still seeks to the wrong location, as before,

Why is there now a discrepancy between what I see in my test page and your player. Your player is now working worse, for some reason.

Thanks

Liz

robwalch commented 3 years ago

I am unable to reproduce the media error in v1.0.0. If there is a specific environment that produces a media error in v1.0.0 please log a new issue focused on creating that error and we can prioritize that.

For the seek issues, I can perform the two seeks above in Chrome and Firefox with maxBufferHole and maxFragLookUpTolerance set to 0 and I get:

Chrome/Firefox: 1st seek to 2319.84 seeks to frame 55676 Chrome: 2nd seek to 2309.84 seeks to 55436 Firefox: 2nd seek to 2309.84 seeks to 55440 (repeating the currentTime = 2309.84 updates the rendered frame (but not currentTime) to 55436)

This does not appear to be an issue hls.js can resolve as Firefox is determining to render the nearest available frame and completing the seek before hls.js can load and append the media. The fact that you can do a follow-up seek is promising if you are in need of a workaround. You might consider adding a listener for Events.BUFFER_APPENDED which when video.paused will do video.currentTime = video.currentTime (or time last seeked to) to refresh the decoded/rendered frame.

If you feel that there is a change hls.js can make to fix a certain type of behavior please file a new issue with a narrower area of focus. I apologize for not being able to help more generally with frame accuracy across browsers, but the scope of the issue makes it difficult to suss out what is a browser issue, what we can workaround internally or you might in your implementation, and what is a critical error most end users would notice that we should prioritize over everything else.

SnapStreamJason commented 3 years ago

Starting with Lizzie's test link - https://d25uzy1v1dityp.cloudfront.net/hls_seek_issue.html - using latest Firefox on Windows.

Seek to 55436, shows 55436, then seek to 55196 and it shows 55200, then seek to 55196 again and it shows 55196.

If this is really a browser issue, then it should show up regardless of player used, so I quickly mocked up the same sample in video.js

https://snapstream-dev-test.s3.amazonaws.com/hls_seek_issue_with_videojs_7.10.0.html

https://snapstream-dev-test.s3.amazonaws.com/hls_seek_issue_with_videojs_current.html

With the 7.10.0 url, you'll see that in FireFox you can seek to 55436, shows 55436, then seek to 55196 and it shows 55196, seek again and it still shows 55196.

With the current url, you'll see that in Firefox you can seek to 55436, shows 55436, then seek to 55196, and it shows 55201, seek again and it shows 55196.

Three different behaviors depending on the javascript library used (including one that works perfectly) lead me to believe it is not a browser issue.

robwalch commented 3 years ago

Hi @SnapStreamJason ,

Three different behaviors depending on the javascript library used (including one that works perfectly) lead me to believe it is not a browser issue.

~Can you make the test pages display the buffered ranges so that it can be confirmed that the range being seeked to is not buffered first? The behavior I described is based on an observation using the Timeline tab in hls.js's demo page. When seeking to an unbuffered area that is near a buffered one (4 frames away), the browser is rendering the nearest frame. If you can make the video.js test page prove that it (1) is not buffering the range with 55200 prior to seeking or (2) is buffering on after "seeking" and only "seeked" after the frame is buffered and rendered, then I would say it's on hls.js to improve the behavior.~

videojs 7.10.0 clears the buffer whenever seeking outside of the buffered range. I confirmed this by checking video.buffered.length and .buffered.end(0) after each seek. By removing the forward buffer before the second seek it cannot display 55201 because it's not there anymore.

If this is the behavior you want with hls.js, you can flush the buffer on video "seeking" using hls.trigger(Events.BUFFER_FLUSHING).

robwalch commented 1 year ago

4956 has brought up the topic of seeking back frame-by-frame.

As long as maxBufferHole and maxFragLookUpTolerance are set to 0 or your lowest allowed tolerance for frame accuracy, with these changes HLS.js would load the previous segment so that the correct frame could be decoded and displayed even if the playhead position is within the jagged start of the following segment https://github.com/video-dev/hls.js/compare/bugfix/nudging-over-buffer-hole-when-paused

0sten commented 1 year ago

Hi! Thank you for your great tool. I'm not sure I have to create a separate issue for my problem. I've got similar issue with seeking stuck after a few seeks to same time 122.82. Reproduced in both chrome and firefox on mac, on latest hls.js and dev. here is the link to demo with the stream and here is the config:

{
    "startFragPrefetch": true,
    "debug": true,
    "lowLatencyMode": true,
    "startPosition":122.82,
    "backBufferLength": 0,
    "maxBufferLength": 5,
    "maxBufferSize": 1000000,
}

To reproduce you can just run the script in the console

const seekFrame = () => {
    scheduler
       .postTask(() => seekFrame(), { delay: 2200 });
    const video = document.getElementById('video');
    if (video.paused) {video.play()}
    video.currentTime = 122.82;
}

after few seeks it got stuck

image

Setting maxFragLookUpTolerance to 0 solves the issue. Can I leave it to be always 0? Thank you in advance.