HLS in separate audio and video always out of sync

cgjh1 commented 6 months ago

Have you read the FAQ and checked for duplicate open issues? yes

If the problem is related to FairPlay, have you read the tutorial?

not fairplay problem

What version of Shaka Player are you using? https://cdnjs.cloudflare.com/ajax/libs/shaka-player/4.7.9/shaka-player.compiled.js also tested on demo page https://shaka-player-demo.appspot.com

Can you reproduce the issue with our latest release version? yes

Can you reproduce the issue with the latest code from main? yes

Are you using the demo app or your own custom app? tried both

If custom app, can you reproduce the issue using our demo app? yes

What browser and OS are you using? Tested on Chrome and Safari

For embedded devices (smart TVs, etc.), what model and firmware version are you using? no embedded involved

What are the manifest and license server URIs? It fails on all hls when audio is not in the same file as the video. If you set : Prefer native HLS playback when available = true and play on safari it works, but on chrome hls with audio and video seperat, always out of sync. Not ONE single example found online showing use of hls live with audio/video separate Can not use dash since no player can recover if a live feed stops and the restarts, must have discontinuity support.

What configuration are you using? What is the output of player.getConfiguration()?

What did you do?

What did you expect to happen?

What actually happened?

joeyparrish commented 6 months ago

It fails on all hls when audio is not in the same file as the video. Not ONE single example found online showing use of hls live with audio/video separate

There are several built into the demo app:

https://shaka-player-demo.appspot.com/demo/#panel=ALL_CONTENT;panelData=HLS

For example:

All of these have separate audio and video streams, and all of these are in sync for me. I checked each one in Chrome. Do you get different results?

cgjh1 commented 6 months ago

Hi

Thank you so much for the response, I really appreciate it, however I checked all the links you sent me and the first 3 are clearly marked VOD, in their segmentplaylist:

for example:

if you do :

curl https://storage.googleapis.com/shaka-demo-assets/bbb-dark-truths-hls/hls.m3u8

you get

#EXTM3U
## Generated with https://github.com/google/shaka-packager version 
v2.3.0-5bf8ad5-release
#EXT-X-MEDIA:TYPE=AUDIO,URI="playlist_a-eng-0128k-aac-2c.mp4.m3u8",GROUP-ID="default-audio-group",LANGUAGE="en",NAME="stream_0",AUTOSELECT=YES,CHANNELS="2"
#EXT-X-STREAM-INF:BANDWIDTH=255636,AVERAGE-BANDWIDTH=212901,CODECS="avc1.42c01e,mp4a.40.2",RESOLUTION=250x144,AUDIO="default-audio-group"
playlist_v-0144p-0100k-libx264.mp4.m3u8

..... etc now look at the playlist :playlist_v-0144p-0100k-libx264.mp4.m3u8

curl https://storage.googleapis.com/shaka-demo-assets/bbb-dark-truths-hls/playlist_v-0144p-0100k-libx264.mp4.m3u8

you get:

#EXTM3U
#EXT-X-VERSION:6
## Generated with https://github.com/google/shaka-packager version 
v2.3.0-5bf8ad5-release
#EXT-X-TARGETDURATION:5
#EXT-X-PLAYLIST-TYPE:VOD.  <--------- this is not live
#EXT-X-MAP:URI="v-0144p-0100k-libx264-init.mp4"
#EXTINF:4.000,
v-0144p-0100k-libx264-s1.mp4
#EXTINF:4.000,
v-0144p-0100k-libx264-s2.mp4

... etc

you will get the same for 2 more an the last look also like a vod since its segment number starts on 0 and its an. older version 3 hls list

So as far as I can tell none of the examples you gave are live, that is, none of them start reading from the end of the playlist. Vod work for me as well,

To reproduce my problem is really easy, you can take any source, lets say a video with audio, let ffmpeg loop it, to produce udp in 1x speed using -re in the input, that goes forevever on that loop. Then start shaka packager to generate a live hls manifest with one video one audio streamplaylist and the audio will never be in sync when playing the hls manifest on chrome.

Here is how you can test it:

1.Download a test source:

wget https://storage.googleapis.com/gtv-videos-bucket/sample/TearsOfSteel.mp4

2.Loop it to an udp stream to simulate a running live feed (source is already h264 and aac so no conversion is needed):

ffmpeg -re -stream_loop -1 -i TearsOfSteel.mp4 -c:a copy -c:v copy -f mpegts udp://239.255.50.36:5020

3.check that you have udp data being produced by doing:

ffprobe udp://239.255.50.36:5020

Now you should get:

Input #0, mpegts, from 'udp://239.255.50.36:5020':
   Duration: N/A, start: 11.105944, bitrate: N/A
   Program 1
     Metadata:
       service_name    : Service01
       service_provider: FFmpeg
   Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), 
yuv420p(progressive), 1280x534, 24 fps, 24 tbr, 90k tbn
   Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 
44100 Hz, stereo, fltp, 194 kb/s

All is good

4 Time for the shaka packager, download it here (I am running on linux, I hope you do the same):

wget https://github.com/shaka-project/shaka-packager/releases/download/v2.6.1/packager-linux-x64
chmod +x packager-linux-x64
./packager-linux-x64 --version
packager-linux-x64 version v2.6.1-634af65-release

Nice the packager is working.

Start packager to create a live HLS manifest that we then will play in shaka player:

MYHTMLPATH="/var/www/html/mytest"

./packager-linux-x64 \
'in=udp://239.255.50.36:5020? reuse=1, stream=audio,playlist_name=audio.m3u8, segment_template=$MYHTMLPATH/audio_$Time$.ts' \
'in=udp://239.255.50.36:5020? reuse=1, stream=video,playlist_name=video_sd.m3u8,segment_template=$MYHTMLPATH/video_sd_$Time$.ts' \
--mpd_output /cryptoguard/content/startrek/master.mpd \
--hls_master_playlist_output $MYHTMLPATH/master.m3u8 \
--hls_playlist_type LIVE --time_shift_buffer_depth 120  \
--preserved_segments_outside_live_window 5 --segment_duration 10

Done

If you play this in shaka player demo as a custom asset on chome using the master.m3u8 it will not be in sync

Again thank you so much for looking into this, please let me know if anything is unclear and I will try to explain better.

Regards

Jörgen

roberto-sartori-gl commented 3 months ago

I think I have the same issue: I have a bunch of hls and a bunch of dash streams.

All the dash streams are fine, but all the hls streams show the out of sync problem mentioned here. The hls manifests have separate tracks for audio and video.

bazeport-jhe commented 3 months ago

I can report that I am experiencing what I think is the same issue as described above. AV out of sync with separate audio and video streams. Safari is OK, Chrome and Firefox not OK. Tested with v4.7.11 and 4.8.1

Update: Setup a two test applications one with Shaka player and one with HLS.js - HLS.js plays the content fine in all browsers, Shaka struggles as mentioned above. Another thing to note is that in Shaka the audio is behind almost exactly 3 seconds.

roberto-sartori-gl commented 3 months ago

I think this may be related to this: https://github.com/shaka-project/shaka-player/issues/5482#issuecomment-1811376979

I have some HLS tracks with a certain availability window (3 hours): when I set the configs described in the linked message, and I go back in my availability window, the audio and the video are in sync. This however breaks the 'live' for me, as 'live' does not work anymore (black screen, no audio).

Without the changes in the configs, the live mode works but with out of sync audio.

avelad commented 1 month ago

In order for us to investigate or fix this issue, we need a stream (url) with this case, if you don't want to share it publicly, you can send it to shaka-player-maintainers@googlegroups.com Thank you!

cgjh1 commented 1 month ago

Hi

You can reproduce it by following the instructions in the bug description, it creates a looping udp stream simulationg a live source.

Below is a copy of those instructions:

=========================

Here is how you can test it: **** 1.Download a test source: wget https://storage.googleapis.com/gtv-videos-bucket/sample/TearsOfSteel.mp4 https://storage.googleapis.com/gtv-videos-bucket/sample/TearsOfSteel.mp4%602.Loop it to an udp stream to simulate a running live feed (source is already h264 and aac so no conversion is needed): ffmpeg -re -stream_loop -1 -i TearsOfSteel.mp4 -c:a copy -c:v copy -f mpegts udp://239.255.50.36:5020 3.check that you have udp data being produced by doing: ffprobe udp://239.255.50.36:5020 Now you should get: Input #0, mpegts, from 'udp://239.255.50.36:5020': Duration: N/A, start: 11.105944, bitrate: N/A Program 1 Metadata: service_name : Service01 service_provider: FFmpeg Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1280x534, 24 fps, 24 tbr, 90k tbn Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 194 kb/s All is good 4 Time for the shaka packager, download it here (I am running on linux, I hope you do the same): wget https://github.com/shaka-project/shaka-packager/releases/download/v2.6.1/packager-linux-x64chmod +x packager-linux-x64 ./packager-linux-x64 --version packager-linux-x64 version v2.6.1-634af65-release Nice the packager is working. 5. Start packager to create a live HLS manifest that we then will play in shaka player: MYHTMLPATH="/var/www/html/mytest" ./packager-linux-x64 \ 'in=udp://239.255.50.36:5020? reuse=1, stream=audio,playlist_name=audio.m3u8, segment_template=$MYHTMLPATH/audio_$Time$.ts' \ 'in=udp://239.255.50.36:5020? reuse=1, stream=video,playlist_name=video_sd.m3u8,segment_template=$MYHTMLPATH/video_sd_$Time$.ts' \ --mpd_output /cryptoguard/content/startrek/master.mpd \ --hls_master_playlist_output $MYHTMLPATH/master.m3u8 \ --hls_playlist_type LIVE --time_shift_buffer_depth 120 \ --preserved_segments_outside_live_window 5 --segment_duration 10 Done If you play this in shaka player demo as a custom asset on chome using the master.m3u8 it will not be in sync

Regards

Jorgen

On 2024-07-04 12:51, Álvaro Velad Galván wrote:

In order for us to investigate or fix this issue, we need a stream (url) with this case, if you don't want to share it publicly, you can send it to @.*** Thank you!

— Reply to this email directly, view it on GitHub https://github.com/shaka-project/shaka-player/issues/6277#issuecomment-2208678047, or unsubscribe https://github.com/notifications/unsubscribe-auth/BAVRZGO6TOKNXLUQ2M5CSJ3ZKUSJZAVCNFSM6AAAAABDUPRLYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBYGY3TQMBUG4. You are receiving this because you authored the thread.Message ID: @.***>

--------------L6vqmtXaY8lAY5GfUYQCwwZt Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 8bit

<!DOCTYPE html>

Hi

You can reproduce it by following the instructions in the bug description, it creates a looping udp stream simulationg a live source.

Below is a copy of those instructions:

=========================

Here is how you can test it: ******************************** 1.Download a test source: `wget https://storage.googleapis.com/gtv-videos-bucket/sample/TearsOfSteel.mp4` 2.Loop it to an udp stream to simulate a running live feed (source is already h264 and aac so no conversion is needed): `ffmpeg -re -stream_loop -1 -i TearsOfSteel.mp4 -c:a copy -c:v copy -f mpegts udp://239.255.50.36:5020` 3.check that you have udp data being produced by doing: `ffprobe udp://239.255.50.36:5020` Now you should get: ``` Input #0, mpegts, from 'udp://239.255.50.36:5020': Duration: N/A, start: 11.105944, bitrate: N/A Program 1 Metadata: service_name : Service01 service_provider: FFmpeg Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1280x534, 24 fps, 24 tbr, 90k tbn Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 194 kb/s ``` All is good 4 Time for the shaka packager, download it here (I am running on linux, I hope you do the same): ``` wget https://github.com/shaka-project/shaka-packager/releases/download/v2.6.1/packager-linux-x64 chmod +x packager-linux-x64 ./packager-linux-x64 --version packager-linux-x64 version v2.6.1-634af65-release ``` Nice the packager is working. 5. Start packager to create a live HLS manifest that we then will play in shaka player: ``` MYHTMLPATH="/var/www/html/mytest" ./packager-linux-x64 \ 'in=udp://239.255.50.36:5020? reuse=1, stream=audio,playlist_name=audio.m3u8, segment_template=$MYHTMLPATH/audio_$Time$.ts' \ 'in=udp://239.255.50.36:5020? reuse=1, stream=video,playlist_name=video_sd.m3u8,segment_template=$MYHTMLPATH/video_sd_$Time$.ts' \ --mpd_output /cryptoguard/content/startrek/master.mpd \ --hls_master_playlist_output $MYHTMLPATH/master.m3u8 \ --hls_playlist_type LIVE --time_shift_buffer_depth 120 \ --preserved_segments_outside_live_window 5 --segment_duration 10 ``` Done If you play this in shaka player demo as a custom asset on chome using the master.m3u8 it will not be in sync

Regards

Jorgen

On 2024-07-04 12:51, Álvaro Velad Galván wrote:

In order for us to investigate or fix this issue, we need a stream (url) with this case, if you don't want to share it publicly, you can send it to Thank you!

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.

--------------L6vqmtXaY8lAY5GfUYQCwwZt--

avelad commented 1 month ago

And I ask the same thing again, please I can't put together that entire setup, if you can send the url I would appreciate it, otherwise I won't be able to work on this issue.

cgjh1 commented 1 month ago

Hi

I sent test urls to the @.*** email address

Jorgen

On 2024-07-04 13:14, Álvaro Velad Galván wrote:

And I ask the same thing again, please I can't put together that entire setup, if you can send the url I would appreciate it, otherwise I won't be able to work on this issue.

— Reply to this email directly, view it on GitHub https://github.com/shaka-project/shaka-player/issues/6277#issuecomment-2208714944, or unsubscribe https://github.com/notifications/unsubscribe-auth/BAVRZGIRGK26SE3CG5QRK7TZKUU7XAVCNFSM6AAAAABDUPRLYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBYG4YTIOJUGQ. You are receiving this because you authored the thread.Message ID: @.***>

--------------hPwA9qG13HEJOivdyLeQIzm5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 8bit

<!DOCTYPE html>

Hi

I sent test urls to the email address

Jorgen

On 2024-07-04 13:14, Álvaro Velad Galván wrote:

And I ask the same thing again, please I can't put together that entire setup, if you can send the url I would appreciate it, otherwise I won't be able to work on this issue.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.

--------------hPwA9qG13HEJOivdyLeQIzm5--

avelad commented 1 month ago

@cgjh1 I can reproduce the issue with your HLS URL, I will investigate what happens.

cgjh1 commented 1 month ago

Excellent, thank you!

On 2024-07-04 15:00, Álvaro Velad Galván wrote:

@cgjh1 https://github.com/cgjh1 I can reproduce the issue with your HLS URL, I will investigate what happens.

— Reply to this email directly, view it on GitHub https://github.com/shaka-project/shaka-player/issues/6277#issuecomment-2208929173, or unsubscribe https://github.com/notifications/unsubscribe-auth/BAVRZGNPNUHRGU6VXRFUPHDZKVBOTAVCNFSM6AAAAABDUPRLYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBYHEZDSMJXGM. You are receiving this because you were mentioned.Message ID: @.***>

--------------ipT0iE0qTpIQBhaEIxv0OBcv Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 8bit

<!DOCTYPE html>

Excellent, thank you!

On 2024-07-04 15:00, Álvaro Velad Galván wrote:

I can reproduce the issue with your HLS URL, I will investigate what happens.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

--------------ipT0iE0qTpIQBhaEIxv0OBcv--

avelad commented 1 month ago

It's a bit strange, I have made a PR (https://github.com/shaka-project/shaka-player/pull/6987) that fixes A/V sync problems, but in your case it still fails (the desynchronization is minor , but it keeps happening... )

cgjh1 commented 1 month ago

HI

I will test it wuth a newer version of the shaka packager, and see if that helps, the one used for the test urls is pretty old.

On 2024-07-05 08:37, Álvaro Velad Galván wrote:

It's a bit strange, I have made a PR (#6987 https://github.com/shaka-project/shaka-player/pull/6987) that fixes A/V sync problems, but in your case it still fails (the desynchronization is minor , but it keeps happening... )

— Reply to this email directly, view it on GitHub https://github.com/shaka-project/shaka-player/issues/6277#issuecomment-2210275247, or unsubscribe https://github.com/notifications/unsubscribe-auth/BAVRZGOBCFJAIGPZJSBQB73ZKY5KFAVCNFSM6AAAAABDUPRLYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJQGI3TKMRUG4. You are receiving this because you were mentioned.Message ID: @.***>

--------------iElMPnrX2oCAdipmSN77VVlZ Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 8bit

<!DOCTYPE html>

HI

I will test it wuth a newer version of the shaka packager, and see if that helps, the one used for the test urls is pretty old.

On 2024-07-05 08:37, Álvaro Velad Galván wrote:

It's a bit strange, I have made a PR (#6987) that fixes A/V sync problems, but in your case it still fails (the desynchronization is minor , but it keeps happening... )

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

--------------iElMPnrX2oCAdipmSN77VVlZ--

avelad commented 1 month ago

@cgjh1 Do you have any update?

shaka-bot commented 1 month ago

Closing due to inactivity. If this is still an issue for you or if you have further questions, the OP can ask shaka-bot to reopen it by including @shaka-bot reopen in a comment.

roberto-sartori-gl commented 3 weeks ago

@cgjh1 did you test this? I still have the issue with the latest version of the player.

shaka-project / shaka-player

HLS in separate audio and video always out of sync #6277