search

google / ExoPlayer

This project is deprecated and stale. The latest ExoPlayer code is available in https://github.com/androidx/media
https://developer.android.com/media/media3/exoplayer
Apache License 2.0
21.74k stars 6.03k forks source link

HLS I-frame only tracks detection #8533

Open davydovvladimir opened 3 years ago

davydovvladimir commented 3 years ago

Hi all, I have found that #EXT-X-I-FRAME-STREAM-INF variants are not presented in track groups if CODECS and RESOLUTION attributes are missed. Here is an example of master playlist:

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=3200000,PROGRAM-ID=1,RESOLUTION=1920x1080
https://a787201989-zabava-htvod.cdn.ngenix.net:443/hls/hd_2020_iskusstvennyy_intellekt__16_ar2_trailer/bw3200000/video.ts/playlist.m3u8
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=3200000,URI="https://a787201989-zabava-htvod.cdn.ngenix.net:443/hls/hd_2020_iskusstvennyy_intellekt__16_ar2_trailer/bw3200000/video.ts/iframes.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=2400000,PROGRAM-ID=1,RESOLUTION=1280x720
https://a787201989-zabava-htvod.cdn.ngenix.net:443/hls/hd_2020_iskusstvennyy_intellekt__16_ar2_trailer/bw2400000/video.ts/playlist.m3u8
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=2400000,URI="https://a787201989-zabava-htvod.cdn.ngenix.net:443/hls/hd_2020_iskusstvennyy_intellekt__16_ar2_trailer/bw2400000/video.ts/iframes.m3u8"

This playlist can be found at following url: https://zabava-htvod.cdn.ngenix.net/hls/hd_2020_iskusstvennyy_intellekt__16_ar2_trailer/variant.m3u8 I hope it is accessible from any location but not sure.

In the code of HlsMediaPeriod.java I see following conditions to mark variant as video track: https://github.com/google/ExoPlayer/blob/03263db378392385e290c40505d80e990c85a5cb/library/hls/src/main/java/com/google/android/exoplayer2/source/hls/HlsMediaPeriod.java#L567

As you can see by these conditions height parameter must be present or type of codec should be video. In my case these parameters are unknown. So, i-frame variants are not presented in video tracks. I checked that these parameters corresponds to RESOLUTION and CODECS HLS attributes.

But it seems they are not mandatory attributes because HLS RFC has following description:

Every EXT-X-STREAM-INF tag SHOULD include a CODECS attribute.

Here is SHOULD but isn't MUST.

The RESOLUTION attribute is OPTIONAL but is recommended if the Variant Stream includes video.

RESOLUTION also looks like not mandatory parameters.

Simple fix for this case is to add additional condition to mark it as video if format has trick play flag set. Possible code can looks like this:

if (format.height > 0 || Util.getCodecsOfType(format.codecs, C.TRACK_TYPE_VIDEO) != null || (format.roleFlags & C.ROLE_FLAG_TRICK_PLAY) != 0) {

What do you think?

davydovvladimir commented 3 years ago

Hi, Is any additional info required? Here is EventLogger output received on r2.12.3:

02-04 12:11:00.735  3167  3167 D EventLogger: videoEnabled [eventTime=0.94, mediaPos=0.00, window=0, period=0]
02-04 12:11:00.736  3167  3167 D EventLogger: audioEnabled [eventTime=0.94, mediaPos=0.00, window=0, period=0]
02-04 12:11:00.737  3167  3167 D EventLogger: tracks [eventTime=0.94, mediaPos=0.00, window=0, period=0
02-04 12:11:00.738  3167  3167 D EventLogger:   MediaCodecVideoRenderer [
02-04 12:11:00.738  3167  3167 D EventLogger:     Group:0, adaptive_supported=YES [
02-04 12:11:00.738  3167  3167 D EventLogger:       [X] Track:0, id=0, mimeType=video/avc, bitrate=3200000, res=1920x1080, supported=YES
02-04 12:11:00.738  3167  3167 D EventLogger:       [X] Track:1, id=2, mimeType=video/avc, bitrate=2400000, res=1280x720, supported=YES
02-04 12:11:00.738  3167  3167 D EventLogger:     ]
02-04 12:11:00.738  3167  3167 D EventLogger:     Metadata [
02-04 12:11:00.738  3167  3167 D EventLogger:       HlsTrackMetadataEntry
02-04 12:11:00.738  3167  3167 D EventLogger:     ]
02-04 12:11:00.738  3167  3167 D EventLogger:   ]
02-04 12:11:00.738  3167  3167 D EventLogger:   MediaCodecAudioRenderer [
02-04 12:11:00.739  3167  3167 D EventLogger:     Group:0, adaptive_supported=N/A [
02-04 12:11:00.739  3167  3167 D EventLogger:       [X] Track:0, id=1/15, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=en, supported=YES
02-04 12:11:00.739  3167  3167 D EventLogger:     ]
02-04 12:11:00.739  3167  3167 D EventLogger:   ]
02-04 12:11:00.739  3167  3167 D EventLogger:   TextRenderer [
02-04 12:11:00.739  3167  3167 D EventLogger:     Group:0, adaptive_supported=N/A [
02-04 12:11:00.739  3167  3167 D EventLogger:       [ ] Track:0, id=1/8219, mimeType=application/cea-608, supported=YES
02-04 12:11:00.739  3167  3167 D EventLogger:     ]
02-04 12:11:00.739  3167  3167 D EventLogger:   ]
02-04 12:11:00.739  3167  3167 D EventLogger:   MetadataRenderer [
02-04 12:11:00.740  3167  3167 D EventLogger:     Group:0, adaptive_supported=N/A [
02-04 12:11:00.740  3167  3167 D EventLogger:       [X] Track:0, id=null, mimeType=application/id3, supported=YES
02-04 12:11:00.740  3167  3167 D EventLogger:     ]
02-04 12:11:00.740  3167  3167 D EventLogger:   ]
02-04 12:11:00.740  3167  3167 D EventLogger:   CameraMotionRenderer []
02-04 12:11:00.740  3167  3167 D EventLogger: ]

With my fix the output is following:

02-04 12:20:54.334  3259  3259 D EventLogger: videoEnabled [eventTime=1.22, mediaPos=0.00, window=0, period=0]
02-04 12:20:54.336  3259  3259 D EventLogger: audioEnabled [eventTime=1.22, mediaPos=0.00, window=0, period=0]
02-04 12:20:54.337  3259  3259 D EventLogger: tracks [eventTime=1.22, mediaPos=0.00, window=0, period=0
02-04 12:20:54.337  3259  3259 D EventLogger:   MediaCodecVideoRenderer [
02-04 12:20:54.337  3259  3259 D EventLogger:     Group:0, adaptive_supported=YES [
02-04 12:20:54.338  3259  3259 D EventLogger:       [X] Track:0, id=0, mimeType=video/avc, bitrate=3200000, res=1920x1080, supported=YES
02-04 12:20:54.338  3259  3259 D EventLogger:       [ ] Track:1, id=1, mimeType=video/avc, bitrate=3200000, supported=YES
02-04 12:20:54.338  3259  3259 D EventLogger:       [X] Track:2, id=2, mimeType=video/avc, bitrate=2400000, res=1280x720, supported=YES
02-04 12:20:54.338  3259  3259 D EventLogger:       [ ] Track:3, id=3, mimeType=video/avc, bitrate=2400000, supported=YES
02-04 12:20:54.338  3259  3259 D EventLogger:     ]
02-04 12:20:54.338  3259  3259 D EventLogger:     Metadata [
02-04 12:20:54.338  3259  3259 D EventLogger:       HlsTrackMetadataEntry
02-04 12:20:54.338  3259  3259 D EventLogger:     ]
02-04 12:20:54.338  3259  3259 D EventLogger:   ]
02-04 12:20:54.338  3259  3259 D EventLogger:   MediaCodecAudioRenderer [
02-04 12:20:54.338  3259  3259 D EventLogger:     Group:0, adaptive_supported=N/A [
02-04 12:20:54.338  3259  3259 D EventLogger:       [X] Track:0, id=1/15, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=en, supported=YES
02-04 12:20:54.338  3259  3259 D EventLogger:     ]
02-04 12:20:54.338  3259  3259 D EventLogger:   ]
02-04 12:20:54.338  3259  3259 D EventLogger:   TextRenderer [
02-04 12:20:54.338  3259  3259 D EventLogger:     Group:0, adaptive_supported=N/A [
02-04 12:20:54.339  3259  3259 D EventLogger:       [ ] Track:0, id=1/8219, mimeType=application/cea-608, supported=YES
02-04 12:20:54.339  3259  3259 D EventLogger:     ]
02-04 12:20:54.339  3259  3259 D EventLogger:   ]
02-04 12:20:54.339  3259  3259 D EventLogger:   MetadataRenderer [
02-04 12:20:54.339  3259  3259 D EventLogger:     Group:0, adaptive_supported=N/A [
02-04 12:20:54.339  3259  3259 D EventLogger:       [X] Track:0, id=null, mimeType=application/id3, supported=YES
02-04 12:20:54.339  3259  3259 D EventLogger:     ]
02-04 12:20:54.339  3259  3259 D EventLogger:   ]
02-04 12:20:54.339  3259  3259 D EventLogger:   CameraMotionRenderer []
02-04 12:20:54.339  3259  3259 D EventLogger: ]
davydovvladimir commented 3 years ago

One more thing I forgot. To test it on specified url: https://zabava-htvod.cdn.ngenix.net/hls/hd_2020_iskusstvennyy_intellekt__16_ar2_trailer/variant.m3u8 you need to set UserAgent to: RT-STB-FW/1.0.822 (Hi3798MV200, STB122A) exoplayer/2.12.3 Otherwise, you will see some video stub from telecom operator.

christosts commented 3 years ago

I will push a commit with this change, the issue should be auto-updated when the commit is pushed to the dev-v2 branch.

Thanks.

davydovvladimir commented 3 years ago

Hello,

I will push a commit with this change, the issue should be auto-updated when the commit is pushed to the dev-v2 branch.

Do you have any progress with it? Thanks.

christosts commented 3 years ago

I plan to commit the change this week. :)

christosts commented 3 years ago

There is a regression while testing this change. Assume the following playlist sample, which has a video track and an I-Frame track, both without CODECS and RESOLUTION as per spec:

#EXT-X-STREAM-INF:BANDWIDTH=582387
video.m3u8

#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=38728,URI="v2/iframe.m3u8"

As the code is structured at the moment, with the suggested change, it would pick up only the I-Frame track and skip the first (video) track, which is also incorrect. It would require a bigger code change to fix this properly.

In the recommendations from Apple, CODECS and RESOLUTION MUST be included in EXT-X-I-FRAME-STREAM-INF, see section 9.3 and 9.4 here.

That said, although not in par with the spec, the player is compatible with Apple recommendations. We cannot proceed with the suggested fix to avoid regressions. I will keep the issue in the backlog for a bigger fix, but it will likely be low on priority.

davydovvladimir commented 3 years ago

@christosts, I didn't know about these recommendations. Thanks.

stevemayhew commented 2 years ago

@davydovvladimir Yes, these apple recommendations are pretty key to actually getting a stream to work in all players, there is also a tool mediastreamvalidator (MacOS only unfortunately) that we refer our origin server vendors to so they can test their streams.

Without this metadata the iFrame tracks would be subject to "chunk-full" prepare (See Santiago's Chunkless Prepare article in order to determine if HEVC, AVC, or whatever other video codec was required just to play them. That is, they would need to be probed

@christosts I'll check what the existing code to see if they are actually probed for this. On my public good todo list is to add code to DefaultTrackSelector.Parameters to set this mode and code in the main demo to support using it (enable when the scrub bar is selected)