SSA-in-MKV doesn't support simultaneous subtitles

yuroyami commented 2 years ago

ExoPlayer Version

2.17.1

Devices that reproduce the issue

All devices I used. It's not device-relevant.
Devices I used: OnePlus 8T, OnePlus 10 Pro, Xiaomi Mi11 Ultra, Asus ROG Phone 4, Red Magic 6, Xiaomi Redmi Note 8 Pro, Samsung Galaxy S21
All of the aforementioned devices run Android 10 to 12.

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Yes

Reproduction steps

Loading any media file.
Passing an SRT/ASS subtitle file to ExoPlayer that contains two (or more) lines that show up at the same time.

Expected result

Both subtitles should appear at the same time.

Actual result

If two subtitles are overlapping, the subtitle that appears later will remove the existence of the subtitle that appeared before it, albeit both co-existing during that timeframe.
In many movies and series, subtitles overlap in some dialogues.

Media

OverlappingSubsDemo.zip

As for the video file, just grab any media from https://thedigitaltheater.com/dts-trailers/

Bug Report

[ ] You will email the zip file produced by adb bugreport to dev.exoplayer@gmail.com after filing this issue.

icbaker commented 2 years ago

For SRT (subrip), this is a duplicate of https://github.com/google/ExoPlayer/issues/4794.

For SSA/ASS, this should be supported since https://github.com/google/ExoPlayer/issues/6320 and there are tests that check this (and are passing). Can you please provide an SSA file that reproduces the issue on 2.17.1?

google-oss-bot commented 2 years ago

Hey @chromaticnoob. We need more information to resolve this issue but there hasn't been an update in 14 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

google-oss-bot commented 2 years ago

Since there haven't been any recent updates here, I am going to close this issue.

@chromaticnoob if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.

yuroyami commented 2 years ago

ExampleFiles.zip

Regardless to MimeTypes such as ASS and SRT. There are two categories of subtitles I am talking about: Externally-loaded ones (manually loaded) and internal ones (coming in a MKV container).

The issue does NOT exist with manually-loaded SSA/ASS subtitle files. Here's where the problem is:

SRT subtitle tracks whether they're manually loaded or internal in a MKV container.
and ASS/SSA subtitle tracks that come in a MKV container (Internal subs). (only one sub shows up at a time).

I have attached the following files to reproduce the problem:

3 Subtitle Files: SSA, ASS and SRT - Produced by Aegisub and they all have identical subtitle lines.
2 Video files: 1 MP4 sample and 1 MKV sample that contains 2 subtitle tracks (One is SRT, the other is ASS).

Tested as of version 2.18.0. @icbaker @google-oss-bot I hope this helps.

icbaker commented 2 years ago

Thanks for clarifying that this specifically affects SSA inside MKV - that would explain why the commit I was referencing above isn't 'working'.

And thanks for providing example media, that will help to check any fix made for this works as expected.

The timestamps for SSA subtitles inside Matroska containers are extracted from the Matroska TimeStamp and BlockDuration elements. The MatroskaExtractor then emits a separate ExoPlayer 'sample' for each subtitle. This is in contrast to the way 'sideloaded' SSA subtitles are handled, where the whole file is emitted as a single ExoPlayer 'sample'

The changes in https://github.com/google/ExoPlayer/pull/6595 assume that all the overlapping subtitles appear in the same 'sample'.

We have some much larger subtitle changes in our future plans, which will change how subtitles map to 'samples', and should make the code paths between sideloaded SSA and SSA-in-MKV more similar. I'm afraid don't think it's worth trying to fix this issue with the current code, because it's all going to change, and so the work would largely be wasted - but I will add it to the list of issues we should aim to resolve as part of the much larger refactor. Thanks for raising it :)

icbaker commented 1 month ago

Media3 1.4.0 (starting with alpha02) now parses subtitles earlier on in playback, which resolves the problem of one cue replacing another when they overlap in time - so now both cues are shown. Unfortunately when I tried the sample file from https://github.com/google/ExoPlayer/issues/10295#issuecomment-1178132212 above, the subtitles are drawn directly on top of each other.

This is a similar problem to the WebVTT case described here (but worse, because I think this happens for all SSA cues even with non-explicit 'default' positioning): https://github.com/google/ExoPlayer/issues/10980

google / ExoPlayer