search

androidx / media

Jetpack Media3 support libraries for media use cases, including ExoPlayer, an extensible media player for Android
https://developer.android.com/media/media3
Apache License 2.0
1.56k stars 373 forks source link

In V1.4.0 DVB subtitles not working, also supportsFormat API returns wrong value #1655

Closed bennettpeter closed 2 weeks ago

bennettpeter commented 2 weeks ago

Version

Media3 1.4.0

More version details

No response

Devices that reproduce the issue

Amazon Fire stick, onn 4K streaming box

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

  1. Api error: Playing a video that has text subtitles and selecting a subtitle.
  2. DVB subtitle problem: Playing a video that has DVB subtitles and selecting a subtitle.

Expected result

  1. Plays with subtitles
  2. Plays with subtitles

Actual result

  1. Error "Subtitle type application/x-media3-cues is not supported.
  2. java.lang.IllegalStateException: Legacy decoding is disabled, can't handle application/dvbsubs samples (expected application/x-media3-cues).

This was working fine in prior versions.

  1. Error in supportsFormat API

The methods DefaultSubtitleParserFactory.supportsFormat() and SubtitleDecoderFactory.DEFAULT.supportsFormat() are inconsistent with the new methodology of decoding subtitles in the extractor. The Format objects returned by TrackGroup.getFormat(ixTrack) have application/x-media3-cues in the sampleMimeType and the actual subtitle type (e.g. text/x-ssa) in the codecs field. However, the two supportsFormat() methods are still checking the sampleMimeType and returning false for subtitle support when the sampleMimeType is application/x-media3-cues.

My work-around is to check for hardcoded application/x-media3-cues or application/cea-608

  1. Unable to use DVB subtitles

My streams are all progressive, based on video files (ts, mkv, mp4, etc) served with http.

I am able to enable subtitles for most types, except for dvbsubs. In the case of dvbsubs, TrackGroup.getFormat(ixTrack) returns a Format object with sampleMimeType of application/dvbsubs and codecs of null. Upon enabling the subtitle, there is an exception java.lang.IllegalStateException: Legacy decoding is disabled, can't handle application/dvbsubs samples (expected application/x-media3-cues).

I attempted to fix this by changing /androidx/media3/extractor/ts/DvbSubtitleReader.java commented out .setSampleMimeType(MimeTypes.APPLICATION_DVBSUBS) replaced with
.setSampleMimeType(MimeTypes.APPLICATION_MEDIA3_CUES) .setCodecs(MimeTypes.APPLICATION_DVBSUBS) This was unsuccessful, java.lang.IllegalStateException: Bundle length is not aligned by 4: 33559567

I prefer not to enable legacy decoding, I can stay on the prior version of exoplayer until it is fixed.

The source code for my application with V1.4.0 is in the media-upgrade branch at https://github.com/bennettpeter/android-MythTV-Leanfront/tree/media-upgrade

I have tried the changes suggested in https://github.com/androidx/media/issues/1644 and others. I originally was using ProgressiveMediaSource.Factory, and I changed to use DefaultMediaSourceFactory as suggested. The subtitles other than dvbsubs were already playing correctly with ProgressiveMediaSource.Factory. The change to DefaultMediaSourceFactory made no difference.

I have tested the following subtitle types:

Text: OK DVB: Fail: java.lang.IllegalStateException: Legacy decoding is disabled, can't handle application/dvbsubs samples (expected application/x-media3-cues). PGS: OK ASS: OK CEA-608: OK: note this works in spite of the fact that in this case sampleMimeType is application/cea-608 and codecs is null

Media

  1. Any video with text, PGS, or ass subtitles
  2. Any video with DVB subtitles

Bug Report

icbaker commented 2 weeks ago

2. Any video with DVB subtitles

Please can you provide a clip that we can use to reproduce this. Please either upload it here or send it to android-media-github@google.com with the subject Issue #1655. Please also update this issue to indicate you’ve done this.

It's not true that all DVB subtitles have an issue with 1.4.0, e.g. the media provided in https://github.com/androidx/media/issues/1621 works on phones (and has an issue on TV/STB devices, but not related to 'Legacy decoding is disabled' error)

bennettpeter commented 2 weeks ago

I have uploaded a short video here: https://drive.google.com/file/d/10-fJIsPH1LawJRdMgLNbkHMf_X9wdowr/view

icbaker commented 2 weeks ago

Are you able to reproduce this issue in the demo app? I built it at 1.4.0 and played the file you provided. When I enable the subtitles I don't see any errors in logcat, and subtitle text appears on screen. It seems everything works.

bennettpeter commented 2 weeks ago

I will try that. I assume the demo app is not using legacy subtitle support?

icbaker commented 2 weeks ago

The demo app is not using legacy subtitle support.

bennettpeter commented 2 weeks ago

The clip plays and the subtitles work with the demo application, as you said

I have found the trigger for this exception. In order to cater for possible closed captions, I pass a List of closedCaptionFormats into the DefaultTsPayloadReaderFactory. 

List<Format> closedCaptionFormats = new ArrayList<>();
closedCaptionFormats.add(
        new Format.Builder()
                .setAccessibilityChannel(1)
                .setSampleMimeType(MimeTypes.APPLICATION_CEA608)
                .build());
closedCaptionFormats.add(
        new Format.Builder()
                .setAccessibilityChannel(2)
                .setSampleMimeType(MimeTypes.APPLICATION_CEA608)
                .build());
TsPayloadReader.Factory payloadReaderFactory
        =  new DefaultTsPayloadReaderFactory(
        0,
        closedCaptionFormats);
TsExtractor ts = new TsExtractor(
        TsExtractor.MODE_SINGLE_PMT,
        0,
        new DefaultSubtitleParserFactory(),
        new TimestampAdjuster(0),
        payloadReaderFactory,
        Settings.getInt("pref_tweak_ts_search_pkts") * TsExtractor.TS_PACKET_SIZE);

Playing the clip, I see a choice of two Closed Captions and one English subtitle. If I select either of the Closed Caption items, it carries on playing as there are no closed captions in the clip. If I select the English subtitle, there is an Exception.

If I comment the two closedCaptionFormats.add(... statemenets above and play the clip, it shows only one subtitle choice, and selecting it works and I see the subtitles.

My applications always adds two CEA608 formats to TS videos in case they have closed captions. Perhaps there is a better way to do this?

icbaker commented 2 weeks ago

Thanks for the extra info. I tried to reproduce this in the demo app at 1.4.0 by changing the code in PlayerActivity.createMediaSourceFactory from:

https://github.com/androidx/media/blob/b01c6ffcb3fca3d038476dab5d3bc9c9f2010781/demos/main/src/main/java/androidx/media3/demo/main/PlayerActivity.java#L318-L323

To:

return new DefaultMediaSourceFactory(/* context= */ this,
    new DefaultExtractorsFactory()
        .setTsSubtitleFormats(
            ImmutableList.of(
                new Format.Builder()
                    .setAccessibilityChannel(1)
                    .setSampleMimeType(MimeTypes.APPLICATION_CEA608)
                    .build(),
                new Format.Builder()
                    .setAccessibilityChannel(2)
                    .setSampleMimeType(MimeTypes.APPLICATION_CEA608)
                    .build())))
    .setDataSourceFactory(dataSourceFactory)
    .setDrmSessionManagerProvider(drmSessionManagerProvider)
    .setLocalAdInsertionComponents(
        this::getClientSideAdsLoader, /* adViewProvider= */ playerView)
    .setServerSideAdInsertionMediaSourceFactory(imaServerSideAdInsertionMediaSourceFactory);

I saw the two 'Unknown' subtitles appear in the track selector, and the 'English' one. I tried selecting all three, and didn't see any playback failures, and I see subtitles appear for the 'English' (DVB) case.

My code change should be equivalent to yours, so I'm not sure what is causing your different experience.

The only 'unknown' difference I can see is what value you're using for Settings.getInt("pref_tweak_ts_search_pkts")? The demo app is using the default value of 600: https://github.com/androidx/media/blob/b01c6ffcb3fca3d038476dab5d3bc9c9f2010781/libraries/extractor/src/main/java/androidx/media3/extractor/ts/TsExtractor.java#L123

bennettpeter commented 2 weeks ago

My default value for pref_tweak_ts_search_pkts is 2600.

I tried my program again and it is working correctly.

What happened is, I was using a deprecated constructor for TsExtractor

            TsExtractor ts = new TsExtractor(
                    TsExtractor.MODE_SINGLE_PMT,
                    new TimestampAdjuster(0),
                    payloadReaderFactory,
                    Settings.getInt("pref_tweak_ts_search_pkts") * TsExtractor.TS_PACKET_SIZE);

If you try this version of the constructor (by deleting the 0 and the new DefaultSubtitleParserFactory() parameters), you get the exception.

I must have noticed the deprecated constructor after doing my analysis and fixed it, not re-tested, and not realizing that fixing it was also fixing the problem. I inadvertently fixed the problem myself without realizing it. Sorry for the trouble.

Will you still look into the Error in supportsFormat API or advise on whether I am doing something wrong?

Thanks for the help.

icbaker commented 2 weeks ago

Thanks for the update.

If you try this version of the constructor (by deleting the 0 and the new DefaultSubtitleParserFactory() parameters), you get the exception.

This is somewhat working as intended - as this constructor doesn't do any transcoding of subtitles during extraction, and therefore is relying on leagcy decoding.

Will you still look into the Error in supportsFormat API or advise on whether I am doing something wrong?

I think this is just a misunderstanding/confusing API. In the new way of handling subtitles, SubtitleParser is invoked during extraction, before the sample queue. It transforms subtitles from (in this case) application/dvbsubs to an internal representation (CuesWithTiming), and the extractor then serializes this into the sample queue in the application/x-media3-cues format using CueEncoder. Therefore the format of the data written to the sample queue (which is returned by TrackGroup.getFormat(ixTrack)) is the format after the subtitles have been transcoded (application/x-media3-cues). It's correct that DefaultSubtitleParserFactory.supportsFormat(..) returns false for this MIME type because, this is effectively the output format for SubtitleParser implementations.

The application/x-media3-cues data is decoded on the Renderer side of the sample queue by CueDecoder.

bennettpeter commented 2 weeks ago

Thank you for the clarification. Also thank you for the great level of support that you and your team provides.