search

google / ExoPlayer

This project is deprecated and stale. The latest ExoPlayer code is available in https://github.com/androidx/media
https://developer.android.com/media/media3/exoplayer
Apache License 2.0
21.72k stars 6.03k forks source link

No audio output when CMAF & Multi-Key & Segmented mp4 are packaged. #10364

Open coooltype opened 2 years ago

coooltype commented 2 years ago

Same question as #10136.

  1. No audio when I package Widevine content as [multi-key + Segmented mp4] and play it in Android Exoplayer (2.16.1). But if I package it as [multi-key + Single mp4] it outputs audio. Why is there no audio output for content packaged as [multi-key + segmented mp4] in Exoplayer? Is there a way to output audio with this content?

  2. In Exoplayer, segmented-VTT subtitles are not displayed. Exactly, only the first segmented subtitles are displayed, not later. How can I get segmented-VTT subtitles to be displayed?

[multi-key + Segmented mp4] content information. When I download and play content in Exoplayer Demo, I have no audio.

[multi-key + Segmented mp4] content.
When I download and play content in Exoplayer Demo, I have no audio.

{
        "name": "segmented/cmaf",
        "uri": "https://contents.pallycon.com/DEMO/app/tears_of_steel-segmented/cmaf/stream.mpd",
        "drm_scheme": "widevine",
        "drm_license_uri": "https://license.pallycon.com/ri/licenseManager.do",
        "drm_key_request_properties": {
          "pallycon-customdata-v2": "eyJrZXlfcm90YXRpb24iOmZhbHNlLCJyZXNwb25zZV9mb3JtYXQiOiJvcmlnaW5hbCIsInVzZXJfaWQiOiJ1dGVzdCIsImRybV90eXBlIjoid2lkZXZpbmUiLCJzaXRlX2lkIjoiREVNTyIsImhhc2giOiJwYUoybFVTYmQ5aEpGQ3VkVWNLbFV2WDdaRGZSa3l6VmY2VHBzaTd2VUtvPSIsImNpZCI6ImRlbW8tdG9zLWFkdmFuY2VkIiwicG9saWN5IjoiOVdxSVdrZGhweFZHSzhQU0lZY25Kc2N2dUE5c3hndWJMc2QrYWp1XC9ib21RWlBicUkreGFlWWZRb2Nja3Z1RWZ4RGNjbTdjV2RWWHFyZE1nQVFqbXFmVVhja1doNEgwNGFMODlUa0hKOXUxWjJTUUlhSWFUXC9rd09JUFQyaWZMN2NkK0pBK2l0clpzaHNqbXpxR0R6NWVzOVhtbk0rWktUNnF4WUtOM2o0ekV3WURvTHlBeUhTZzVvN3BVQjVZa1YiLCJ0aW1lc3RhbXAiOiIyMDIyLTA0LTAxVDA5OjA5OjE5WiJ9"
        }
}
I would also like to know why the subtitles are not displayed.

I would also like to know why the subtitles are not displayed.

icbaker commented 2 years ago

I tried playing the provided stream in the 2.16.1 demo app and I observe that playback never advances beyond 0:00. I see the following lines in logcat that might be relevant:

An exception:

2022-06-21 09:28:09.273 22692-22751/com.google.android.exoplayer2.demo E/TrackGroup:   java.lang.IllegalStateException: Different languages combined in one TrackGroup: 'en-stereo-aac' (track 0) and 'null' (track 1)
        at com.google.android.exoplayer2.source.TrackGroup.logErrorMessage(TrackGroup.java:188)
        at com.google.android.exoplayer2.source.TrackGroup.verifyCorrectness(TrackGroup.java:154)
        at com.google.android.exoplayer2.source.TrackGroup.<init>(TrackGroup.java:58)
        at com.google.android.exoplayer2.source.dash.DashMediaPeriod.buildPrimaryAndEmbeddedTrackGroupInfos(DashMediaPeriod.java:680)
        at com.google.android.exoplayer2.source.dash.DashMediaPeriod.buildTrackGroups(DashMediaPeriod.java:512)
        at com.google.android.exoplayer2.source.dash.DashMediaPeriod.<init>(DashMediaPeriod.java:147)
        at com.google.android.exoplayer2.source.dash.DashMediaSource.createPeriod(DashMediaSource.java:583)
        at com.google.android.exoplayer2.source.MaskingMediaPeriod.createPeriod(MaskingMediaPeriod.java:128)
        at com.google.android.exoplayer2.source.MaskingMediaSource.onChildSourceInfoRefreshed(MaskingMediaSource.java:203)
        at com.google.android.exoplayer2.source.MaskingMediaSource.onChildSourceInfoRefreshed(MaskingMediaSource.java:38)
        at com.google.android.exoplayer2.source.CompositeMediaSource.lambda$prepareChildSource$0$com-google-android-exoplayer2-source-CompositeMediaSource(CompositeMediaSource.java:115)
        at com.google.android.exoplayer2.source.CompositeMediaSource$$ExternalSyntheticLambda0.onSourceInfoRefreshed(Unknown Source:4)
        at com.google.android.exoplayer2.source.BaseMediaSource.refreshSourceInfo(BaseMediaSource.java:84)
        at com.google.android.exoplayer2.source.dash.DashMediaSource.processManifest(DashMediaSource.java:962)
        at com.google.android.exoplayer2.source.dash.DashMediaSource.onManifestLoadCompleted(DashMediaSource.java:739)
        at com.google.android.exoplayer2.source.dash.DashMediaSource$ManifestCallback.onLoadCompleted(DashMediaSource.java:1395)
        at com.google.android.exoplayer2.source.dash.DashMediaSource$ManifestCallback.onLoadCompleted(DashMediaSource.java:1390)
        at com.google.android.exoplayer2.upstream.Loader$LoadTask.handleMessage(Loader.java:475)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loopOnce(Looper.java:201)
        at android.os.Looper.loop(Looper.java:288)
        at android.os.HandlerThread.run(HandlerThread.java:67)

Lots of lines like this (2 is C.TRACK_TYPE_VIDEO):

com.google.android.exoplayer2.demo E/BaseMediaChunkOutput: Unmatched track of type: 2

The track selection output shows the 3 webvtt tracks have been detected. However it also shows the audio track has been identified as a video track, and grouped into the same track group with the other video tracks:

tracks [eventTime=11.81, mediaPos=0.00, window=0, period=0
  MediaCodecVideoRenderer [
    Group:0, adaptive_supported=NO [
      [ ] Track:0, id=3, mimeType=audio/mp4a-latm, bitrate=204756, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=44100, language=en-stereo-aac, supported=NO
      [X] Track:1, id=4, mimeType=video/avc, bitrate=4219719, codecs=avc1.640028, drm=[playready,widevine,cenc], res=1920x800, fps=24.0, supported=YES
      [X] Track:2, id=5, mimeType=video/avc, bitrate=2219395, codecs=avc1.4d401f, drm=[playready,widevine,cenc], res=1280x534, fps=24.0, supported=YES
      [X] Track:3, id=6, mimeType=video/avc, bitrate=1123908, codecs=avc1.4d401e, drm=[playready,widevine,cenc], res=854x356, fps=24.0, supported=YES
    ]
  ]
  MediaCodecAudioRenderer []
  TextRenderer [
    Group:0, adaptive_supported=N/A [
      [X] Track:0, id=0, mimeType=text/vtt, bitrate=416, language=en, supported=YES
    ]
    Group:1, adaptive_supported=N/A [
      [ ] Track:0, id=1, mimeType=text/vtt, bitrate=463, language=de, supported=YES
    ]
    Group:2, adaptive_supported=N/A [
      [ ] Track:0, id=2, mimeType=text/vtt, bitrate=376, language=it, supported=YES
    ]
  ]
  MetadataRenderer []
  CameraMotionRenderer []
]

I took a look at the MPD file and I noticed that there is one audio AdaptationSet and 2 video ones. Both of the video ones have SupplementalProperty nodes with adaptation-set-switching. Simplified, it looks like this:

<AdaptationSet id="3" contentType="audio" segmentAlignment="true" lang="eng_stereo_aac">
  ...
</AdaptationSet>
<AdaptationSet id="4" contentType="video" maxWidth="1920" maxHeight="800" frameRate="90000/3750" par="12:5">
  ...
  <SupplementalProperty schemeIdUri="urn:mpeg:dash:adaptation-set-switching:2016" value="3"/>
  ...
</AdaptationSet>
<AdaptationSet id="5" contentType="video" width="854" height="356" frameRate="90000/3750" segmentAlignment="true" par="12:5">
  ...
  <SupplementalProperty schemeIdUri="urn:mpeg:dash:adaptation-set-switching:2016" value="4"/>
  ...
</AdaptationSet>

This is basically saying that set 4 (video) can be seamlessly switched with set 3 (audio), which is nonsense.

If I hack ExoPlayer to ignore the adaptation-set-switching properties then it groups the tracks to match the AdaptationSet and the content plays successfully.

tracks [eventTime=0.33, mediaPos=0.00, window=0, period=0
  MediaCodecVideoRenderer [
    Group:0, adaptive_supported=YES [
      [X] Track:0, id=4, mimeType=video/avc, bitrate=4219719, codecs=avc1.640028, drm=[playready,widevine,cenc], res=1920x800, fps=24.0, supported=YES
      [X] Track:1, id=5, mimeType=video/avc, bitrate=2219395, codecs=avc1.4d401f, drm=[playready,widevine,cenc], res=1280x534, fps=24.0, supported=YES
    ]
    Group:1, adaptive_supported=N/A [
      [ ] Track:0, id=6, mimeType=video/avc, bitrate=1123908, codecs=avc1.4d401e, drm=[playready,widevine,cenc], res=854x356, fps=24.0, supported=YES
    ]
  ]
  MediaCodecAudioRenderer [
    Group:0, adaptive_supported=N/A [
      [X] Track:0, id=3, mimeType=audio/mp4a-latm, bitrate=204756, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=44100, language=en, supported=YES
    ]
  ]

To fix this you should fix your manifest so it doesn't declare streams as seamlessly switchable if they're not.

Once playback was working I selected the subtitle tracks manually and didn't see any subtitles appear. I haven't looked into what's happening there.

Aside: The manifest also seems to use an invalid value for the language of the audio track:

<AdaptationSet id="3" contentType="audio" segmentAlignment="true" lang="eng_stereo_aac">

The DASH spec (ISO 23009-1:2014 section 5.3.3.2) defines the lang attribute as:

The syntax and semantics according to IETF RFC 5646 shall be used.

RFC 5646 does not permit a string of the form eng_stereo_aac (syntax defintion).

coooltype commented 2 years ago

Thanks for the detailed explanation on problem 1. However, 2 problems remain. Request you to check.

  1. In Exoplayer, segmented-VTT subtitles are not displayed. Exactly, only the first segmented subtitles are displayed, not later. How can I get segmented-VTT subtitles to be displayed?
[multi-key + Segmented mp4] content.
{
        "name": "segmented/cmaf",
        "uri": "https://contents.pallycon.com/DEMO/app/tears_of_steel-segmented/cmaf/stream.mpd",
        "drm_scheme": "widevine",
        "drm_license_uri": "https://license.pallycon.com/ri/licenseManager.do",
        "drm_key_request_properties": {
          "pallycon-customdata-v2": "eyJrZXlfcm90YXRpb24iOmZhbHNlLCJyZXNwb25zZV9mb3JtYXQiOiJvcmlnaW5hbCIsInVzZXJfaWQiOiJ1dGVzdCIsImRybV90eXBlIjoid2lkZXZpbmUiLCJzaXRlX2lkIjoiREVNTyIsImhhc2giOiJwYUoybFVTYmQ5aEpGQ3VkVWNLbFV2WDdaRGZSa3l6VmY2VHBzaTd2VUtvPSIsImNpZCI6ImRlbW8tdG9zLWFkdmFuY2VkIiwicG9saWN5IjoiOVdxSVdrZGhweFZHSzhQU0lZY25Kc2N2dUE5c3hndWJMc2QrYWp1XC9ib21RWlBicUkreGFlWWZRb2Nja3Z1RWZ4RGNjbTdjV2RWWHFyZE1nQVFqbXFmVVhja1doNEgwNGFMODlUa0hKOXUxWjJTUUlhSWFUXC9rd09JUFQyaWZMN2NkK0pBK2l0clpzaHNqbXpxR0R6NWVzOVhtbk0rWktUNnF4WUtOM2o0ekV3WURvTHlBeUhTZzVvN3BVQjVZa1YiLCJ0aW1lc3RhbXAiOiIyMDIyLTA0LTAxVDA5OjA5OjE5WiJ9"
        }
}
I would also like to know why the subtitles are not displayed.