Closed kakao-wise-kim closed 1 year ago
The most recent main branch does not reproduce this behavior. There are a few PRs related to the audio buffer that have been addressed, but it looks like these will be released to the new 1.5 version. Hopefully we'll see a release in the near future.
Could this gave been fixed by #5742?
@robwalch
What version should I use to use the version with #5742 applied? This PR looks like it's supposed to apply to 1.5. Has it been applied to 1.4.11-Canary-9471 by any chance? I still have the same symptoms in 1.4.11-Canary.
The reason I ask is that I'm using hls.js from npm and combine it with external modules, so it's very easy to test im my test environments if you know the version.
Hi @kakao-wise-kim,
Yes, dev and the latest canary has it:
https://0f22780e.hls-js-dev.pages.dev/demo/ (v1.4.11-0.canary.9471)
Hi @robwatch
This version also made a similar action. But It has not same symptoms with a version 1.4.2 to 1.4.10. This version has try to download normal time audio segments when a network throttling situation.
But, video segments download action was not normal. It was try to download video segments what a high level. And, It was canceled. and seeked playback timeline. And try downed level video segment downloading again. But It also maybe high level on a throttle network states. When this process is repeated, playback stops and only video segments are downloaded in the background.
After playback stops, playback will not start even if you click the Play button. It have to refresh to get back to normal.
What kind of throttling are you performing exactly?
@robwalch Use the Network Condition tool that exists in chrome development tools. The live stream consists of 1 audio and 5 video levels. Most of the issues occur when throttling occurs sharply. For example, when I set the Network Condition from No Throttling to 750k.
@robwalch This afternoon I ran more tests and saw the same symptoms as before.
From my analysis, the symptoms are as follows
I increased the TTL of the CDN to avoid the 404, and here are the results of my tests.
The time of requested for audio download here varied from case to case: 10 seconds, 20 seconds, 30 seconds, over a minute, etc.
I have a question here.
The most recent main branch does not reproduce this behavior. There are a few PRs related to the audio buffer that have been addressed, but it looks like these will be released to the new 1.5 version. Hopefully we'll see a release in the near future.
v1.5.0 will be in beta in the coming weeks, with a stable release some time in September.
Why does it only download the video segment first, after a certain amount of buffering, even though it is live?
Audio segment loading is handled by the audio-stream-controller doTickIdle()
method. Audio segments should be loaded along side video segments (from media `currentTime), but they should not be loaded more than one segment or target duration past the end the video buffer.
Is it possible to set the number of video segments or time to download first? It only downloads video segments up to a minute or more. I'd like to limit the amount of time that only videos are downloaded when buffering etc. occurs, is there an option I can set?
HLS.js fills the main (video) forward buffer to at least maxBufferLength
and idles only after maxMaxBufferLength
or maxBufferSize
has been reached (whichever comes first).
The audio buffer should follow the end of the video buffer +1 segment within the same discontinuity sequence.
Version 1.4.1 downloads audio and video together, are you thinking of changing this to be the same?
All versions should perform this way. It is still unclear why that is not the case with your live stream - but only for certain versions from your perspective. If this is fixed in dev, then the issue is resolved.
Looking at your screenshots, the issue is with your audio playlist not updating. If your audio playlist is behind, there is no way for the player to load segments it does not know about. This could be an issue with playlist requests timing out, bad target duration values, misaligned media timestamps, or something else - we can't tell without a sample.
I have tested 1.4.10 with live streams that I have access to and throttles down to 750kbps/100ms-rtt with the lowest level also being around 500-600kbps and I could not reproduce what you are describing. The player may have stalled for a couple of segments as the average causes level 1 to be selected (second lowest) before lowest, but audio and video were in lock step.
@robwalch
hello. I made a mistake when I explained about reproduction testing earlier. The audio is not 1 level. It is 3 levels. Video consists of 5 levels.
I found repeated one symptom here. is that if I use only one audio, I won't have the problem. However, if I configure multiple audio levels, the problem is reproduced when switching. This problem is also happening in the recently updated version 1.4.12.
This is our Master Playlist Sample.
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="a1",NAME="a0",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",LANGUAGE="ko",URI="a0_a1/a1.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="a2",NAME="a0",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",LANGUAGE="ko",URI="a0_a2/a2.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="a3",NAME="a0",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",LANGUAGE="ko",URI="a0_a3/a3.m3u8"
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=3192000,BANDWIDTH=6384000,CODECS="avc3.640020,mp4a.40.2",RESOLUTION=1280x720,FRAME-RATE=60.000,AUDIO="a1"
v0_v3/v3.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=8192000,BANDWIDTH=16384000,CODECS="avc3.64002a,mp4a.40.2",RESOLUTION=1920x1080,FRAME-RATE=60.000,AUDIO="a1"
v0_v1/v1.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=1128000,BANDWIDTH=2256000,CODECS="avc3.64001f,mp4a.40.2",RESOLUTION=854x480,FRAME-RATE=30.000,AUDIO="a2"
v0_v5/v5.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=628000,BANDWIDTH=1256000,CODECS="avc3.64001e,mp4a.40.2",RESOLUTION=640x360,FRAME-RATE=30.000,AUDIO="a2"
v0_v7/v7.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=264000,BANDWIDTH=528000,CODECS="avc3.640015,mp4a.40.2",RESOLUTION=428x240,FRAME-RATE=30.000,AUDIO="a3"
v0_v8/v8.m3u8
If a Buffer Stall error occurs while audio level switching is occurring, the switched audio pts are set olded time the video pts. If the audio fragments continue to download and looks similar to the video pts, playback will start.
This symptom does not occur every time. Often they will switch normally. The symptom is mainly reproduced when a buffer stall occurs with level switching, as shown below.
What version of Hls.js are you using?
1.4.2 to 1.4.10
What browser (including version) are you using?
chrome 116.0.5845.96
What OS (including version) are you using?
mac os 13.4.1
Test stream
If you need a link, please leave your email address.
Configuration
Additional player setup steps
I can see this in the hls.js demo, so there's nothing to set up.
Checklist
Steps to reproduce
I can reproduce the symptom when playing adaptive live videos with audio and video separated (m4s files) and multiple quality.
Step1 - Run the live video in a normal environment. Play for a sufficiently long time (enough time for a file of more than 9 segments to be downloaded and played). Step2 - After a period of time, set a random delay on network. I usually use the Network Conditions option in Chrome browser. Step3 - Limit the Network Condition until you encounter "Type: 'networkError', details: 'fragLoadError'" or "type: 'mediaError', details: 'bufferStalledError'" Buffer as an error. (Repeat the normal and limited settings, but don't do it too fast)
Expected behaviour
The video should switch to a lower quality and play.
What actually happened?
In this case, Video File tries to download a file in normal time, but Audio File tries to download a old time file. As a result, playback stops, and if you try to retry playback, it doesn't work. Therefore, the video buffer will continue to fill first. The Audio buffer is filled, but the time difference with the Video buffer is very large. Or, Audio buffer is not filled. Sometimes It try to download audio files from a long time ago. If the file doesn't exist due to a CDN TTL timeout, you'll get a 404.
Console output
Chrome media internals output