AV freeze indefinitely in HLS stream having multiple #EXT-X-PROGRAM-DATE-TIME with jump in segments with discontinuity.

[Lohit-Huddar]

EXOPLAYER_VERSION = com.amazon.android:exoplayer:2.11.3 exoplayer source code: https://github.com/amzn/exoplayer-amazon-port/tree/amazon/r2.11.3 Tested on demo app from "exoplayer amazon port".

I see AV freeze and no player errors thrown when i stream windowed live HLS manifest with variant having multiple #EXT-X-PROGRAM-DATE-TIME with jump in timestamp, below bolded text is the snippet of variant playlist having jump in timestamp.

EXT-X-PROGRAM-DATE-TIME:2022-01-17T13:10:10.424Z

EXTINF:2.002,

http://hostname_xyz/cmaf-hls/ad-1_001.cmfv

EXTINF:2.002,

http://hostname_xyz/cmaf-hls/ad-1_002.cmfv

EXT-X-DISCONTINUITY

EXT-X-MAP:URI="https://hostname/video/vid05/init.mp4"

EXT-X-PROGRAM-DATE-TIME:2022-01-17T13:15:22.161Z

EXTINF:0.192,

https://hostname/video/vid05/1856.m4s

Same variant works fine on iOS and seeing issue only in android. Also, if its playing content which has second #EXT-X-PROGRAM-DATE-TIME, i was expecting window.windowStartTimeMs to have second #EXT-X-PROGRAM-DATE-TIME but after going through https://github.com/google/ExoPlayer/issues/8312 and going through source code of exoplayer, i see that multiple #EXT-X-PROGRAM-DATE-TIME is not handled and playlist time is window start time which is populated during variant playlist download. For me, even activePlayer.getCurrentPosition() is not returning proper value, in this case, getCurrentPosition() should return after 5mins jump when its playing 1856.m4s.

I will send mail to dev.exoplayer@gmail.com with charles dump, adb logs as well stream details with this ticket number which can help for local repro.

[christosts]

Thank you for sending us the additional information on email.

Quick check: have you tried playing the stream on the latest ExoPlayer version, which at the moment is 2.16.1? You are using the amazon fork but I see that it is updated with 2.16.1.

I quickly tried the stream with vanilla ExoPlayer 2.16.1 and it plays fine until I got a 403 (I have not investigated on the 403s more yet).

[Lohit-Huddar]

Thanks @christosts for quick check and reply :) As you suggested, i tried on latest exoplayer version 2.16.1, no luck, issue exists (Source code: https://github.com/google/ExoPlayer/releases/tag/r2.16.1) Also, i tried on amazon port exoplayer version 2.16.1, no luck there as well ( Source code: https://github.com/amzn/exoplayer-amazon-port/blob/amazon/r2.16.1) Device i tried: Amazon Fire stick 3rd Gen, model: AFTSS Repro steps: Wait for 3mins of streaming and we see AV freeze.

I have dropped mail to dev.exoplayer@gmail.com with adb logs and stream details, please try and let me know the status, Thanks !

[christosts]

When I try to play the content you have provided us, after loading the first minute of content, a request to a specific segment is failing with HTTP 403. While this error is logged in the demo app, I do not see it in the logs that you provided to us, so I assume that's not the problem you are facing. It's likely a restriction in how I can play this content. Do you know if the content is geo-protected?

[Lohit-Huddar]

@christosts Correct, i am able to play till 3mins and its AV freeze after that, we need to be in US network for this stream to work.

[christosts]

The stream is now returning HTTP 502, including using a US-based IP address. Can you please provide a fresh link and is it possible to let it alive for longer?

Thanks

[Lohit-Huddar]

@christosts Its DRM protected short lived content in production env, we cant change validity since it impacts field users, can we have google meet call at your convenient time? That will help to easy repro and helps in understanding problem i am facing.

[christosts]

Can you please create some content that stays alive for 48-72 hours, and if yes, please create early in the week (=> Monday-Tuesday)? Now that we sorted out that the content is geo-restricted, there shouldn't be other delays in testing.

[Lohit-Huddar]

Hi @christosts, I have dropped mail with details for content which lasts for 48 to 72 hours, please quickly retry and let me know if it works, Thanks!

[christosts]

Thanks, I can repro now. It's somewhat similar to #8312 and maybe related to #8959.

Let me write up the what's happening for visibility. This is a live playlist. As the player refreshes the playlist, at some point the latest playlist snapshot starts with an EXT-X-PROGRAM-DATE-TIME that has a gap compared to the previous playlist snapshot. In the example snippet on the issue description, this happens when the downloaded playlist starts with #EXT-X-PROGRAM-DATE-TIME:2022-01-17T13:15:22.161Z, where the gap is about 5 minutes. The starting position of segments is defined relative to the first EXT-X-PROGRAM-DATE-TIME of the playlist. Segments will now have a starting point relative to 13:15:22.161Z and have a 5 minute gap from previous segments. At this point, the player assumes that there are 5 minutes of data buffered, because the buffer duration is defined startTimeOfNextSegmentToDownload - currentPosition. The player thinks that's plenty of buffered data and never starts a new download.

I'll try to experiment with short changes to see if we can fix this by shifting the PTS of segments to be relative to the EXT-X-PROGRAM-DATE-TIME before the gap.

I might ask for another stream link, if the current one expires.

Thanks

[christosts]

Update: @Lohit-Huddar will you be able to apply the attached patch on r2.16.1 and try with your streams? It's working with the test link you provided us but maybe better to test on your side too

candidate-fix-9872.patch.zip

[Lohit-Huddar]

Thanks @christosts for quick fix, i tried patch on 2.16.1 tag (https://github.com/google/ExoPlayer/releases/tag/r2.16.1) and still see AV freeze for sample stream i had shared. Below is the code patch and AV freeze frame screenshots. Attached is the adb logs, have printed playlist starttime and durations with tag "Mylog"

![image](https://user-images.githubusercontent.com/98005513/152283961-125751e8-e322-47af-a0d7-81b9

9b833e28.png)

AdbLogs.txt

[christosts]

Yep,I saw similar freezes too in tests after I posted the patch. Unfortunately, a proper fix requires more changes (should be addressed together with #8312 and #8959). We'll keep this issue open to track the fix, but I'm afraid I can't provide a timeline of when we'll work on this.

In the meantime, my advice is to try to eliminate these timestamp jumps in the playlist. It could be (note: not tested) that the symptom faced in this case does not happen if the timestamp jumps are smaller than the max buffer duration set in LoadControl (by default, that's DefaultLoadControl.DEFAULT_MAX_BUFFER_MS which is 50 seconds).

[stevemayhew]

Thanks, I can repro now. It's somewhat similar to #8312 and maybe related to #8959.

This is most definitely a duplicate of #8312. This HLS coding is attempting to replicate the functionality of EXT-X-GAP without using that tag.

Also, jumps greater than 50 seconds will most definitely lead to stalls, this was the bug I reported in #8959. We are running with this fix https://github.com/google/ExoPlayer/pull/9050 in our fork of ExoPlayer to solve this issue (believe it or not this happens more than you would think in live feeds from satellite)

[Darkos-den]

hi @tianyif , any updates here? I encountered a similar problem with the thread hanging. Shouldn't #EXT-X-DISCONTINUITY cause the timestamps to be recalculated? So the second #EXT-X-PROGRAM-DATE-TIME is simply ignored?