Support SSA/ASS styling

[jakobkukla]

Add support for SSA/ASS styling. Currently there is only "minimal" support for ASS subtitles since styling was removed from the initial PR. (See https://github.com/google/ExoPlayer/pull/2281)

It would be great to have full support of the spec. Many subtitles use these styling features to work directly with the video (e.g. to substitute text in a video). SSA/ASS styling is also very common in certain genres of media like Cartoons, Anime or Karaoke.

[andrewlewis]

@icbaker Tentatively marking as an enhancement, but please dupe or close as needed. Thanks!

[icbaker]

Thanks for the request.

Unfortunately we don't currently have any capacity to improve ExoPlayer's SSA/ASS styling support. However we would welcome high quality PRs in this area.

I'm also happy to discuss various approaches with PR authors if a new feature doesn't fit easily into the existing code structure (either before the PR, or in the review, whichever is easier).

[13steinj]

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

(I'm not familiar with what exactly even is supported currently).

[icbaker]

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

That sounds like a good idea - the best way to do that is probably to compare the SsaDecoder to a copy of the spec. I've found the SSA/ASS spec to be fairly loose, and I'm not sure there's a canonical definition, but I've generally referred to this when working with these files in the past: http://moodub.free.fr/video/ass-specs.doc (linked from Matroska's page on SSA).

In case it's helpful, we would probably prioritise things roughly in this order:

• Features that look like they're implemented but don't work properly (basically bugs).
• Features that are similar to existing ones, and so can be implemented with small incremental changes.
• Features that don't fit into the existing code architecture - so implementing them requires significant refactoring.

I don't know if SSA has anything in the final category - but if it looks like it does then please come talk to us before spending ages refactoring SsaDecoder, as we might have some ideas about the best way to approach the problem.

[cosmo321]

I'm not a developer, so I can't exactly contribute a lot, but proper support of ass/ssa in Exoplayer is something I'd very much would like to have, as Plex for Android currently makes me sad... I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

[icbaker]

I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

Thanks for the pointer to libass, I took a quick look. I think you're possibly suggesting two alternative things:

• ExoPlayer could depend on libass and use it's logic for all the SSA rendering.
• People working on improving ExoPlayer's existing SSA rendering logic could use libass as a reference implementation to see how different pieces of content should be rendered, as an augmentation to the "spec".

In case anyone is considering trying the first of these, I'd like to pre-emptively warn against it. libass looks like it's written in C, and we're very reluctant to pull in native/JNI dependencies unless a pure Java implementation of the same behaviour is deemed ~impossible. Given SSA/ASS is a relatively simple plaintext-based format, it seems completely feasible to parse it fully using pure Java - so it's extremely unlikely we'd accept a PR that introduced a native dependency to achieve this. There's more discussion about (not) using native deps for parsing subtitles in #7746 and #7749.

The second suggestion (use it as a reference implementation) makes perfect sense - and afaict it's used by VLC, so it should be enough to just play any content with VLC and compare to ExoPlayer's behaviour.

[cosmo321]

I leave that up to actual developers to decide. I just thought I'd point out that libass exists and is probably the best reference we have, and that rendering subs differently from libass could potentially make people like me wonder why the subs behave differently in ExoPlayer. The second alternative is probably the better option for ExoPlayer. :)

[szaboa]

hey @icbaker , I'll check what the current SsaDecoder supports and what is missing in regards of styling, based on the specs you linked above, then I could probably start and implement those step by step (with my limited time).

@jakobkukla did you have anything specific in mind? I mean is there a specific styling option which you would like to see first?

[icbaker]

Thanks for taking a look! Breaking down the work into small PRs definitely sounds good.

[jakobkukla]

@szaboa To be honest I'm not really aware what is currently supported and what not. But the ability to display fonts and colors properly would be nice!

[szaboa]

@icbaker I've checked out and it seems that only the v4+ "alignment" is supported. Here is the list of all the style attributes of v4 and v4+ from the specs. Do you see specific styles that would be very cumbersome to implement at first sight? E.g. thinking of "fontname", and "encoding"..

Style name	v4	v4+	Note	Format line support	Override support	PR
Name	✅	✅	-	✅	-	-
Fontname	✅	✅	-	❌	❌	-
Fontsize	✅	✅	-	✅	❌	#8615
PrimaryColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR	✅	❌	#8490
SecondaryColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR	❌	❌	-
OutlineColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR Called OutlineColour in v4+, and TertiaryColour in v4	❌	❌	-
BackColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR	❌	❌	-
Bold	✅	✅	-	✅	❌	#8654
Italic	✅	✅	-	✅	❌	#8654
Underline	❌	✅	-	✅	❌	#8851
Strikeout	❌	✅	-	✅	❌	#8851
ScaleX	❌	✅	-	❌	❌	-
ScaleY	❌	✅	-	❌	❌	-
Spacing	❌	✅	-	❌	❌	-
Angle	❌	✅	-	❌	❌	-
BorderStyle	✅	✅	-	❌	❌	-
Outline	✅	✅	-	❌	❌	-
Shadow	✅	✅	-	❌	❌	-
BorderStyle	✅	✅	-	❌	❌	-
Alignment	✅	❌	values may be 1=Left, 2=Centered, 3=Right. Add 4 to the value for a "Toptitle". Add 8 to the value for a "Midtitle"	❌	❌	-
Alignment	❌	✅	layout of the numpad (1-3 sub, 4-6 mid, 7-9 top)	✅	✅	-
MarginL, MarginR, MarginV	✅	✅	-	❌	-	#10169
AlphaLevel	❌	✅	-	❌	❌	-
Encoding	✅	✅	-	❌	❌	-

My plan is to start implement some easy ones first :)

[icbaker]

The spec I linked above (https://github.com/google/ExoPlayer/issues/8435#issuecomment-756651245) seems to think that Underline, Strikeout, ScaleX, ScaleY, Spacing and Angle are supported in v4+. In fact are possibly only supported there and not in v4. Maybe those rows in your table are accidentally flipped? Or maybe I'm misreading the spec...

The SsaDecoder currently ignores v4 styling completely. It's not clear if that's sensible - I don't know how prevalent each version is in the wild.

If you're going to add v4 support we probably need to do think about whether we can reuse the SsaStyle class or split it into two. It's up to you, but for now imo it would be easiest to focus on v4+ only.

These all look like they should be relatively easy (I've referenced the relevant Android styling span where it seems obvious):

• Fontname (TypefaceSpan)
• PrimaryColour (ForegroundColorSpan)
• BackColour (BackgroundColorSpan)
• Bold (StyleSpan)
• Italic (StyleSpan)
• Underline (UnderlineSpan)
• Strikeout (StrikethroughSpan)
• AlphaLevel (read together with PrimaryColour, though the spec I'm reading says "SSA does not use this yet" so maybe not worth bothering with)
• MarginL, R and V (in fact there's already a TODO for these :)). These will feed into Cue's position & line fields. This might be a bit messy - suggest you don't start here.

The spec I'm looking at doesn't really define these clearly enough to be sure:

• Fontsize (I suspect you can use RelativeSizeSpan or AbsoluteSizeSpan but I don't know which is relevant)
  • EDIT: Small correction - if we work out how the spec defines this, we can use Cue.textSize and textSizeType for encoding size information from the "Format" section of the file that applies to the whole cue. We only need to use spans for overrides that affect substrings of the cue text.

These look harder. They have basically no support in the existing Cue interchange format:

• SecondaryColour
• OutlineColour
• ScaleX and ScaleY
• Spacing
• Angle
• BorderStyle
• Outline
• Shadow

These are hard for other reasons:

• Encoding - I'm not sure how you're supposed to read this since it's in a text file - so you've already had to guess the encoding (right?). Currently we assume it's always UTF-8.

[szaboa]

Yep, you are right, the Underline, Strikeout, ScaleX, ScaleY, Spacing and Angle are supported in v4+ only, and not in v4. I will correct the table.

Thanks for the tips, will definitely start with the easy ones.

[szaboa]

Initial PR for "Primary Colour" is here #8490.

[moneytoo]

Is there any option/API to disable, customize or detect the styling? If an app sets custom CaptionStyleCompat, it now accepts all styling while the text color is now variable based on line from subtitle file. Also, how about when user uses platform CaptioningManager to customize subtitle look - again, text color is now variable. With static background/border colors and variable text color, the subtitle line may end up low contrast on unreadable (right...?). Does my thinking here make any sense? (I've never used SSA/ASS outside some testing.)

[icbaker]

@moneytoo Assuming you're using the ExoPlayer UI components, you can force the SubtitleView to ignore 'embedded' styles (i.e. those encoded in subtitle files) by calling subtitleView.setApplyEmbeddedStyles(false). You can get the SubtitleView from [playerView.getSubtitleView()](https://exoplayer.dev/doc/reference/com/google/android/exoplayer2/ui/PlayerView.html#getSubtitleView()).

[szaboa]

I just want to note that I haven't forgotten about this, just need to find time to continue it :) Updated the table above with the PrimaryColour support.

[szaboa]

Initial PR for "FontSize" is here: #8615

[viggy96]

I don't understand how this is a 'low priority'. Proper SSA/ASS subtitle support is crucial to a video player.

[Weevild]

What's the latest update? And would this progress and (hopefully) final integration of ass work with the Nvidia Shield TV?

[icbaker]

What's the latest update? And would this progress and (hopefully) final integration of ass work with the Nvidia Shield TV?

The table in the comment above has the latest status (it's been updated with PRs merged since then).

I don't see any reason this wouldn't work on Nvidia Shield.

[szaboa]

I plan to continue with the contributions on this, just haven't had too much free time lately. Is there any preferred priority regarding the styling features?

[cosmo321]

No preference on priority, really. Glad to hear it's not forgotten. :)

[Weevild]

I don't see any reason this wouldn't work on Nvidia Shield.

It works but when using home media services such as Plex a transcode is always guaranteed, I'd assume this is because ASS isn't fully supported. Plex could probably do something about this at their end but that doesn't seem like an active inquiry.

[13steinj]

@Weevild ASS isn't fully supported on Plex for android, ironically, because ExoPlayer doesn't support ASS yet, and Plex devs don't want to go through the effort themselves. It's actually quite silly because they could transcode the ASS subtitles to a different format rather than the entire video, but don't. Even when this issue is resolved on ExoPlayer's end it'll take some time for Plex to pull it in.

[Weevild]

Plex devs don't want to go through the effort themselves.

Not a shocker there, but one could only wish. I switched to emby because they support virtually every format that Plex doesn't (subtitle-wise), but to have native support would just be a relief knowing my media is supported regardless.

transcode the ASS subtitles to a different format rather than the entire video

I think emby even does this too, guess they've got their hands full on that free content.

[albino1]

@sixones what's needed from szaboa's list to get this working well enough to merge ASS support into Plex on Android devices? If there's some obvious priority thing I think it would be beneficial for a lot of people if it gets tackled first.

[sixones]

@albino1 Probably not the right place for this, but Plex on Android does support ASS subtitles without the styling already (be sure to set your subtitle burn option to only image formats). We are working on full ASS support with positioning, styling and animations at the moment. If we transcode them into another subtitle format they would end up being the same as the basic level of support which we already offer - without the burn in option changed we burn them into the video to keep the full ASS styling with animations, styles and positions above the video.

[albino1]

Thanks for the reply, makes perfect sense. Appreciate everything you do!

[an-ghinid]

@sixones is there an issue open on your github to track that? Very keen on avoiding ASS transcoding as it's the only thing keeping me from running PMS entirely on my NAS rather than on my Shield TV (and would keep me paying for Plex Pass for the hardware acceleration support that allows this)

[cosmo321]

We are working on full ASS support with positioning, styling and animations at the moment.

@sixones I would've loved to see some more information about that in one of the relevant threads on the Plex forums. Proper ASS support is so long overdue I can't even.

[fixel99]

+1 to still really needing/wanting support for this

[szaboa]

No promises as have very limited time, but I'll try to look into the Margin L, R, V thingy on the weekend.

[lsl330]

No promises as have very limited time, but I'll try to look into the Margin L, R, V thingy on the weekend.

thank you for continuing to improve the ass's subtitle, in addition, I found that the bilingual (two-line) ass subtitle support also has problems, many times can only show one language (one line)

[szaboa]

PR with Margin L, R, V support coming soon.

[szaboa]

@icbaker PR for MarginL, MarginR, MarginV support: https://github.com/google/ExoPlayer/pull/10169.

[szaboa]

@lsl330

thank you for continuing to improve the ass's subtitle, in addition, I found that the bilingual (two-line) ass subtitle support also has problems, many times can only show one language (one line)

I can't find any information about bilingual subtitles in the SSA/ASS specification, do you have more information on this by any chance? Also could you please provide a sample .ass sub to check it out meanwhile?

[sixones]

I can't find any information about bilingual subtitles in the SSA/ASS specification, do you have more information on this by any chance? Also could you please provide a sample .ass sub to check it out meanwhile?

There's no bilingual support as such in ASS/SSA, but you can show multiple subtitle lines at the same time, usually at the top and bottom of the screen, some files can use this for different languages (usually for song lyrics).

[lsl330]

@szaboa

I can't find any information about bilingual subtitles in the SSA/ASS specification, do you have more information on this by any chance? Also could you please provide a sample .ass sub to check it out meanwhile?

here is the sample ass file，it can show Chinese and japanese subtitles，but in exoplayer，can only show the chinese subtitles。 sample ass.zip

thank you！

[szaboa]

@lsl330 I have just tested the subtitle you provided, basically there are multiple lines with the same or overlapping times, as @sixones mentioned. This is supported in ExoPlayer from version 2.11.0 (2019-12-11). It has been solved here https://github.com/google/ExoPlayer/issues/6320. Which version are you using?

... however, in your case these lines have the same position, which is not handled so they are drawn on top of each other.

The SSA/ASS specs mentions two kind of collision handling, normal and reverse, I'll have a look on this.

[lsl330]

@szaboa

OK,thank you very much!!! I use plex and the exoplayer version is 2.17.1.

[david2tm]

Hi, I'm not sure if it's a right place to write this -- but I want to try some coding related to SSA/ASS. The list of stuff these subtitles can do is extensive. So I started looking at the code (mind it is the first time I'll try coding for Android), but I got env set-up already -- with example of subtitles which are not working as they should (looking at VLC rendering) and I want to make them work. I wander, if it's possible to use WebView to render these Cue's? I see that there's WebViewSubtitleOutput... so I guess it's already something. Am I on the right path?

[david2tm]

@icbaker I looked into what SSA/ASS supports and what is implemented and what not. So what I think about it. Currently there are two "subtitle" renderers. CanvasSubtitleOutput (the default) and WebViewSubtitleOutput (which I'm not sure how it's launched, I guess it's done by the app which uses ExoPlayer). Anyway - to play with it I forced the WebViewSubtitleOutput, just because doing some styling is easier for me personally with CSS. Many features ASS supports are not so easily done with CSS as well. I even played with doing stuff like BorderStyle, Outline, Shadow with SVGs, as CSS doesn't support BorderStyle-Outline the way ASS requires it. And I'm not even talking about Effects -- which are basically animations. I then went further down wanting to check how libass works and came across their project, which uses webassembly. Btw according to libass github page, Crunchyroll is using this project. I even entertained myself with using said webassembly with WebViewSubtitleOutput, but it's a no go. As it stands - ExoPlayer looks at subtitles as a series of cues being rendered on a WebView and/or Canvas. Libass creates just another "video" stream based on description of text file. This makes sense. Check this example page. It showcases some capabilities of SSA/ASS which I'm not sure are at all possible with cues. So what are next steps?

1. One can hope that someone ports libass to Java -- which, probably will never happen :)

2. Quoting you from above. I now think that parsing is indeed easy, but rendering is next to impossible (with cues).

   libass looks like it's written in C, and we're very reluctant to pull in native/JNI dependencies unless a pure Java implementation of the same behaviour is deemed ~impossible. Given SSA/ASS is a relatively simple plaintext-based format, it seems completely feasible to parse it fully using pure Java

3. Keep it as it is -- as in, not ever adding the full list of features.

wdyt?

[buliaoyin]

Re-implement SSA/ASS render would be a pure way, but takes much more time to get full feature support, I think import libass with JNI would be the easiest way to make things done.

[jsravn]

Plex has now side stepped this by bundling libass in their android client, presumably using ffmpeg or something to mux on the client side before passing to exoplayer.

[mugiritsu]

Re-implement SSA/ASS render would be a pure way, but takes much more time to get full feature support, I think import libass with JNI would be the easiest way to make things done.

That would be not only the easiest way, but also the only way to correctly render all those obscure and advanced feature-rich ASS files that typesetting enthusiasts can come up with. There are a lot .ass files with over 25MB of animated signs, and at this point not even those long-standing libass alternatives like vsfilter or its derivates can render them properly. Trying to recreate libass would be a neverending task and a huge magnet for complaints about any kind of advanced signs not rendering correctly. Not to mention, libass has years of performance optimizations.

Just adopting libass instead of making a new subtitle renderer could solve a lot of problems.

[Sparh4wk]

I would love to vouch for this one as well.. this is awesome player, but its rly bad for anime without full support for ssa subtitles :( Please make this happen.

[GlassedSilver]

Re-implement SSA/ASS render would be a pure way, but takes much more time to get full feature support, I think import libass with JNI would be the easiest way to make things done.

That would be not only the easiest way, but also the only way to correctly render all those obscure and advanced feature-rich ASS files that typesetting enthusiasts can come up with. There are a lot .ass files with over 25MB of animated signs, and at this point not even those long-standing libass alternatives like vsfilter or its derivates can render them properly. Trying to recreate libass would be a neverending task and a huge magnet for complaints about any kind of advanced signs not rendering correctly. Not to mention, libass has years of performance optimizations.

Just adopting libass instead of making a new subtitle renderer could solve a lot of problems.

I wholeheartedly agree.

If this issue hadn't been such a long-standing one and been actually looked into by Google itself rather than them watching from the sidelines and hoping for the community to fully do this job by the looks of it, I'd be a lot more optimistic about exoplayer re-implementing and keeping up with changes in-time or mostly in-time. But apparently relying on constantly staying in-sync with ass extensions is a futile effort, so just using libass seems like the most reliable, future-proof-ish effort there is from my limited understanding.

It's a shame, but the dynamic nature of ass is both a curse, but also a blessing because it offers features not found elsewhere. The cost may be annoying. But when libass is doing the work needed why don't we just rely on it like what I think any sensible player does with proper ass-support?

Why re-invent the wheel? It's not like there's much to gain from it right? Even if using libass might not be as performant as a fully-optimized re-implementation, what sorts of differences that can be felt are we talking about here? Subtitle rendering isn't something that needs to scale endlessly, especially not when it's per player, at humanly digestible play-speeds, etc...

To anyone who is investing any sort of work into this effort: you have my most honest gratitude. The day Android has player-independent ACE ass support will be a good day! Not only for us anime and FOSS app fans.

[FongMi]

Is there way to fix overlapping subtitles ?

[GlassedSilver]

Is there way to fix overlapping subtitles ?

It's a known issue, any fix would be supplied by exoplayer extending ass support. I'm not aware of user settings that can fix this.