Lip sync broken if maxLiveSyncPlaybackRate is set rather than 1

wakabayashik commented 1 year ago

What version of Hls.js are you using?

v1.3.3

What browser (including version) are you using?

Google Chrome Version 110.0.5481.78 (Official Build) (64-bit)

What OS (including version) are you using?

Windows 11 Pro, (22000.978)

Test stream

No response

Configuration

{
  "debug": true,
  "backBufferLength": 30,
  "maxMaxBufferLength": 120,
  "maxLiveSyncPlaybackRate": 1.01,
  "liveSyncDurationCount": 2
}

Additional player setup steps

open "https://hls-js.netlify.app/demo/" with google chrome
[Save as (Ctrl+S) > Webpage, Complete (.htm;.html)] into local http server (nginx)

run ffmpeg with following options on local http server to generate HLS files

-y -rtsp_transport tcp -i rtsp://{network-live-encoder-hostname}/{stream-name}
-c:v copy -c:a copy -pes_payload_size 0
-hls_time 2 -hls_list_size 3 -hls_start_number_source epoch
-hls_flags temp_file+delete_segments+discont_start+omit_endlist
-hls_playlist_type 0 -hls_enc 0
-hls_segment_filename "{target-directory}/segment%08d.ts" "{target-directory}/playlist.m3u8"

(ffmpeg version 5.0.1-full_build-www.gyan.dev)

Checklist

[x] The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
[X] The issue occurs in the stable client (latest release) on https://hls-js.netlify.com/demo and not just on my page
[X] The issue occurs in the latest client (main branch) on https://hls-js-dev.netlify.com/demo and not just on my page
[X] The stream has correct Access-Control-Allow-Origin headers (CORS)
[X] There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

open demo page through local http server
enter URL for playlist.m3u8 on local http server, configuration and click [Apply]

Expected behaviour

playback without lip sync shifting

What actually happened?

lip sync is gradually shifting during playback

If maxLiveSyncPlaybackRate is 1, this issue does not occur

Console output

hls.js.download:7445 [log] > [level-controller]: Attempt loading level index 0 with URL-id 0 http://192.168.11.221/pub/test/playlist.m3u8
hls.js.download:1745 [log] > [level-controller]: live playlist 0 REFRESHED 1675926376--1
hls.js.download:1819 [log] > [level-controller]: reload live playlist 0 in 1985 ms
hls.js.download:8536 [log] > [stream-controller]: Level 0 loaded [1675926374,1675926376], cc [0, 0] duration:6.006
hls.js.download:7445 [log] > [level-controller]: Attempt loading level index 0 with URL-id 0 http://192.168.11.221/pub/test/playlist.m3u8
hls.js.download:1745 [log] > [level-controller]: live playlist 0 REFRESHED 1675926377--1
hls.js.download:1819 [log] > [level-controller]: reload live playlist 0 in 1990 ms
hls.js.download:8536 [log] > [stream-controller]: Level 0 loaded [1675926375,1675926377], cc [0, 0] duration:6.005999999999997
hls.js.download:15536 [log] > destroy
hls.js.download:15568 [log] > detachMedia
hls-demo.js.download:40582 Using Hls.js config: {debug: true, enableWorker: true, lowLatencyMode: true, backBufferLength: 30, maxMaxBufferLength: 120, …}backBufferLength: 30debug: trueenableWorker: trueliveSyncDurationCount: 2lowLatencyMode: truemaxLiveSyncPlaybackRate: 1.01maxMaxBufferLength: 120[[Prototype]]: Object
hls.js.download:23643 [log] > Debug logs enabled for "Hls instance"
hls.js.download:15616 [log] > stopLoad
hls.js.download:15584 [log] > loadSource:http://192.168.11.221/pub/test/playlist.m3u8
hls.js.download:8485 [log] > [stream-controller]: Trigger BUFFER_RESET
hls.js.download:15557 [log] > attachMedia
hls.js.download:7197 [log] > [level-controller]: manifest loaded, 1 level(s) found, first bitrate: 0
hls.js.download:3174 [log] > 1 bufferCodec event(s) expected
hls.js.download:15606 [log] > startLoad(-1)
hls.js.download:7536 [log] > [level-controller]: switching to level 0 from -1
hls.js.download:7445 [log] > [level-controller]: Attempt loading level index 0 with URL-id 0 http://192.168.11.221/pub/test/playlist.m3u8
hls.js.download:3002 [log] > [stream-controller]: STOPPED->IDLE
hls.js.download:3002 [log] > [subtitle-stream-controller]: STOPPED->IDLE
hls.js.download:1745 [log] > [level-controller]: live playlist 0 MISSED
hls.js.download:1819 [log] > [level-controller]: reload live playlist 0 in 1000 ms
hls.js.download:8536 [log] > [stream-controller]: Level 0 loaded [1675926375,1675926377], cc [0, 0] duration:6.005999999999999
hls.js.download:2802 [log] > [stream-controller]: Live playlist sliding: 0.00 start-sn: 1675926375->1675926375 prev-sn: na fragments: 3
hls.js.download:3627 [log] > [buffer-controller]: Updating Media Source duration to 6.006
hls.js.download:3085 [log] > [buffer-controller]: Media source opened
hls.js.download:1745 [log] > [level-controller]: live playlist 0 MISSED
hls.js.download:1819 [log] > [level-controller]: reload live playlist 0 in 1000 ms
...

no errors

Chrome media internals output

No response

chloekim02 commented 1 year ago

@wakabayashik were you able to fix this issue ? I am having similar issue but only happens in safari. and when I set maxLiveSyncPlaybackRate:1 there is no issue but it starts to have an issue when it's more than 1.

wakabayashik commented 1 year ago

I have already given up on keeping up live position using the maxLiveSyncPlaybackRate option, and instead use liveMaxLatencyDurationCount to ensure that the playback position is not too far from the live position.

chloekim02 commented 1 year ago

does it solve delay issue without playbackRate though? cos when I move around tab, sometimes it's delayed and I want the video catch up the latency

wakabayashik commented 1 year ago

If liveMaxLatencyDurationCount is specified with maxLiveSyncPlaybackRate=1, the delay from live will gradually increase until liveMaxLatencyDuration, where it will jump back to the live position (or liveSyncDuration) Thus, liveMaxLatencyDurationCount might not be a solution if the goal is to always keep playing back the live position with correct lip-sync.

paxelpixel commented 3 months ago

If liveMaxLatencyDurationCount is specified with maxLiveSyncPlaybackRate=1, the delay from live will gradually increase until liveMaxLatencyDuration, where it will jump back to the live position (or liveSyncDuration) Thus, liveMaxLatencyDurationCount might not be a solution if the goal is to always keep playing back the live position with correct lip-sync.

@wakabayashik I've encountered exactly the same issue, what causes these gradual increase of latency?

wakabayashik commented 3 months ago

the accuracy of the clock generator (crystal oscillator) between the live camera and the playback environment
network jitter on the transmission path
accidental processor spikes in the playback environment
et. al.

paxelpixel commented 3 months ago

the accuracy of the clock generator (crystal oscillator) between the live camera and the playback environment

network jitter on the transmission path

accidental processor spikes in the playback environment

et. al.

it seems like this sliding window when false would sync us into the !liveSyncPosition but there is this other check for media.readyState < 4? any idea why? cause if its 4 it can get us to sync.

robwalch commented 2 weeks ago

This is a Chromium issue that should have been fixed since Chrome version 128: https://issues.chromium.org/issues/40190553

video-dev / hls.js