TS stream pixelating when subtitles are parsed during extraction

okycelt commented 4 weeks ago

Version

Media3 1.4.0

More version details

The first commit with this issue is 0352db9. We weren't able to reproduce with 5c6f48e.

Devices that reproduce the issue

Changhong GIU3A00 (Android 9)
Changhong GIU6770 (Android 9)
SEI Robotics Box R Lite 4K (Android 11)
Amlogic TwinKey2 (Android 7.1.1)
Amino Amigo7x (Android 12)

Devices that do not reproduce the issue

Pixel 3a (Android 12)
Lenovo TB-X606F (Android 10)

Reproducible in the demo app?

Yes

Reproduction steps

Play the file sent via email in Media3 demo app (either from local storage or via UDP).

Parsing subtitles during rendering fixes the issue:

textRenderer.experimentalSetLegacyDecodingEnabled(true)

mediaSourceFactory.experimentalParseSubtitlesDuringExtraction(false)

Expected result

The media plays without pixelating even when subtitles are parsed during extraction

Actual result

The media is pixelating with the following error in the logcat every time there is a pixelation in the video

Audio sink error
androidx.media3.exoplayer.audio.AudioSink$UnexpectedDiscontinuityException: Unexpected audio track timestamp discontinuity: expected 1000029511400, got 1000029799400
    at androidx.media3.exoplayer.audio.DefaultAudioSink.handleBuffer(DefaultAudioSink.java:997)
    at androidx.media3.exoplayer.audio.MediaCodecAudioRenderer.processOutputBuffer(MediaCodecAudioRenderer.java:773)
    at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.bypassRender(MediaCodecRenderer.java:2358)
    at androidx.media3.exoplayer.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:868)
    at androidx.media3.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:1136)
    at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:561)
    at android.os.Handler.dispatchMessage(Handler.java:102)
    at android.os.Looper.loop(Looper.java:193)
    at android.os.HandlerThread.run(HandlerThread.java:65)

Media

Sent via email

Bug Report

[ ] You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

icbaker commented 4 weeks ago

Thank you for reporting. I'm afraid I'm not able to reproduce the issue on my Pixel 7a.

I see the subtitles in your media are application/dvbsubs which is a bitmap-based subtitle format. When subtitles are parsed during rendering we decode to a Bitmap object on a standalone thread. When they are parsed during extraction we originally decode to a Bitmap on the loading thread (which is OK to do heavy work on) and then serialize them to bytes in the SampleQueue, before parsing them back to a Bitmap on the playback thread in the renderer. I'm wondering if the issue you're seeing could be related to time spent decoding the Bitmap on the playback thread.

Do you have a more powerful device you could try on (maybe a phone rather than a TV/STB?), to confirm that you're also unable to repro in that case? This would give us some signal that this only occurs on low-powered devices.

tonihei commented 4 weeks ago

In addition to the question above, could you also describe in more detail what exactly "pixelated" means in this case? That probably helps to understand where the problem is exactly.

Is the video output of a lower resolution than expected?
Does the video output have artefacts (for example green pixels)?
Is something wrong with the timing of the video (e.g. dropped frames, delayed frames, wrong A/V sync)?
Is audio playback working correctly (given your error log about audio discontinuities)?

okycelt commented 3 weeks ago

Do you have a more powerful device you could try on (maybe a phone rather than a TV/STB?), to confirm that you're also unable to repro in that case? This would give us some signal that this only occurs on low-powered devices.

I've just tested on a Pixel 3a (Android 12) and a Lenovo TB-X606F (Android 10) and both devices played without issue. In both cases, I played the content from local storage.

In addition to the question above, could you also describe in more detail what exactly "pixelated" means in this case? That probably helps to understand where the problem is exactly.

Is the video output of a lower resolution than expected?

No

Does the video output have artefacts (for example green pixels)?

Yes, but not green pixels. Please see the attached video https://github.com/user-attachments/assets/aed91369-8509-45e3-8cab-95ae53bd5102

Is something wrong with the timing of the video (e.g. dropped frames, delayed frames, wrong A/V sync)?

There are often dropped frames following the audio sink errors, EventLogger also reports them. A/V sync seems to be correct when the video output has no artefacts.

Is audio playback working correctly (given your error log about audio discontinuities)?

Audio seems to be playing without glitches despite the audio sink errors

okycelt commented 3 weeks ago

If you'd like us to test something, feel free to ask.

androidx / media