search

google / ExoPlayer

This project is deprecated and stale. The latest ExoPlayer code is available in https://github.com/androidx/media
https://developer.android.com/media/media3/exoplayer
Apache License 2.0
21.7k stars 6.02k forks source link

Support different #EXT-X-PROGRAM-DATE-TIME tags for different #EXT-X-DISCONTINUITY sections #8312

Open apptrash opened 3 years ago

apptrash commented 3 years ago

Problem

There is a chance that HLS content provider may produce discontinuities in stream. For example, because the server was restarted. In this case, according to the standard (https://tools.ietf.org/html/draft-pantos-hls-rfc8216bis-08#section-4.4.4.3), there must be #EXT-X-DISCONTINUITY tag in manifest. As far as I can understand, exoplayer will try to correct the position using the PTS. In general, PTS is not the best source of truth for this because after #EXT-X-DISCONTINUITY tag there may be any timestamp (even descending). But even if the PTS monotonously increased during, for example, a server reboot, there is another problem: seeking. Assume, that we have VOD HLS manifest:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-START:TIME-OFFSET=0
#EXT-X-TARGETDURATION:10
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-PROGRAM-DATE-TIME:2020-12-02T18:33:03.447Z
#EXTINF:7.920,
2020-12-02/18/segment_1.ts
#EXT-X-PROGRAM-DATE-TIME:2020-12-02T18:33:10.887Z
#EXTINF:9.080,
2020-12-02/18/segment_2.ts
#EXT-X-DISCONTINUITY
#EXT-X-PROGRAM-DATE-TIME:2020-12-02T19:33:20.567Z
#EXTINF:9.040,
2020-12-02/18/segment_3.ts
#EXT-X-PROGRAM-DATE-TIME:2020-12-02T19:33:29.607Z
#EXTINF:9.720,
2020-12-02/18/segment_4.ts
#EXT-X-ENDLIST

If we play sequentially, without seeking, then the player, based on the PTS, will jump the gap (that is, getCurrentPosition will return the time one hour later). But if we seek after the gap, then the player will not know anything about the gap and will return the time that we specified in seekTo.

Summation: discontinuities in stream in conjunction with seeking lead to incorrect position.

Proposed solution

Take #EXT-X-PROGRAM-DATE-TIME after #EXT-X-DISCONTINUITY into account if present (especially after seeking).

There is a recommendation in standard: https://tools.ietf.org/html/draft-pantos-hls-rfc8216bis-08#section-6.2.1

The server MAY associate an absolute date and time with a Media
Segment by applying an EXT-X-PROGRAM-DATE-TIME tag to it.  This
defines an informative mapping of the (wall-clock) date and time
specified by the tag to the first media timestamp in the segment,
which may be used as a basis for seeking, for display, or for other
purposes. If a server provides this mapping, it SHOULD apply an EXT-X-PROGRAM-DATE-TIME
tag to every segment that has an EXT-X-DISCONTINUITY tag applied to it.

And also:

The Server MUST NOT add any EXT-X-PROGRAM-DATE-TIME tag to a Playlist
that would cause the mapping between program date and Media Segment to become ambiguous.

So it is server-side responsibility to provide correct EXT-X-PROGRAM-DATE-TIME values.

An exemplary algorithm for how this can work

If, during sequential playback, we have reached EXT-X-DISCONTINUITY tag and the next segment starts with EXT-X-PROGRAM-DATE-TIME then correct player's position and duration. Moreover, it is possible to correct duration earlier, when this tags appear in manifest.

During seek, player should set position considering previous EXT-X-DISCONTINUITY, EXT-X-PROGRAM-DATE-TIME tags. For live playlists player can remember these tags and correct position based on them

ExoPlayer version: 2.12.1

alex3d commented 3 years ago

An exemplary algorithm for how this can work

If, during sequential playback, we have reached EXT-X-DISCONTINUITY tag and the next segment starts with EXT-X-PROGRAM-DATE-TIME then correct player's position and duration. Moreover, it is possible to correct duration earlier, when this tags appear in manifest.

During seek, player should set position considering previous EXT-X-DISCONTINUITY, EXT-X-PROGRAM-DATE-TIME tags. For live playlists player can remember these tags and correct position based on them

I don't think that's a good idea because it will confuse window/playlist position and duration with wall clock position/duration based on EXT-X-PROGRAM-DATE-TIME. I suggest: 1) Ignore PTS diff on EXT-X-DISCONTINUITY (and probably any PTS diff that makes the current segment larger than the duration specified in EXTINF). It will make currentPosition consistent whether we have reached it with or without seek and allow an application to map currentPosition to playlist item and relative position inside this item. 2) (optional) Provide helper methods to map currentPosition to/from wall clock time based on EXT-X-PROGRAM-DATE-TIME tags. It should probably be left to the application to decide how to map wall clock time to window time because of potential gaps in wall clock time.

christosts commented 3 years ago

@AquilesCanta can you please evaluate this enhancement suggestion?

apptrash commented 3 years ago

Is there any progress on this task?

AquilesCanta commented 3 years ago

Regarding discontinuities and PTS, ExoPlayer already handles that as you would expect it. If it doesn't you should file a fresh issue including all the info required in the issue template.

Regarding program-date-time: Back when we implemented support for EXT-X-PROGRAM-DATE-TIME the explicit decision was made to drop all program-date-time tags after the first one. In order to change this now, because of the way the ExoPlayer timeline works, we would have to use a different period for each discontinuity section, which is a considerable change.

You can generate your own PROGRAM-DATE-TIME / DISCONTINUITY mappings in the app by looking at the downloaded media playlist at any time (getCurrentManifest) and then checking for the PROGRAM-DATE-TIME tag that applies at your current position. It's a bit cumbersome to do the parsing, but it's doable.

I'll mark this as a low priority enhancement, since you can work around it in the app.

apptrash commented 3 years ago

Regarding discontinuities and PTS, ExoPlayer already handles that as you would expect it.

@AquilesCanta, Do you mean that it handles that like @alex3d suggested?

stevemayhew commented 3 years ago

Santiago (@AquilesCanta) we have one origin vendor that is dealing with live playback restart following transcoder outages in this fashion, basically:

EXT-X-PROGRAM-DATE-TIME:2020-12-02T18:33:10.887Z

EXTINF:9.080,

2020-12-02/18/segment_2.ts

EXT-X-DISCONTINUITY

EXT-X-PROGRAM-DATE-TIME:2020-12-02T19:33:20.567Z

EXTINF:9.040,

2020-12-02/18/segment_3.ts

Is logically the same as:

EXT-X-PROGRAM-DATE-TIME:2020-12-02T18:33:10.887Z

EXTINF:9.080,

2020-12-02/18/segment_2.ts

EXT-X-GAP

EXTINF:3600.92,

missing_segment.ts

EXT-X-DISCONTINUITY

EXTINF:9.040,

2020-12-02/18/segment_3.ts

So it's more then informational, the window duration / timeline and the Chunk.startTimeUs would be affected if we wanted to support seeking in this situation.

Note, with SOCU implementations these "gaps" would be recorded into the rolling buffer (as there is no choice).

I realize this is a pretty large architectural change, perhaps we can inject a gap Chunk for this?

stevemayhew commented 3 years ago

Note, in the live case (where transcoder stops for example do to an outage), if the origin server generates GAP tags this would result in far better UX, eg.

  1. DISCONTINUITY / PDT Solution — player would get a stuck playlist error and retry (that would take 2.5 * EXT-X-TARGETDURATION seconds of buffering state before it is reported. This would continue to until transcode restarts
  2. GAP solution — player would see GAP tag and be able to report this as a UI event right away, event report would continue for each GAP tag (every TARGETDURATION) until live playback continues

Obviously, at least for live, GAP solution has a better UX

AquilesCanta commented 3 years ago

Over to @christosts who's now maintaining HLS.