Track selection: Require certain text role flags

[sobbe81]

Hi,

We have a case where we have a DASH assets that can have combinations of Subtitles in multiple languages and/or Closed captions in multiple languages

Our goal is to achieve the following

1. Use captions only if enabled in the CaptioningManager
2. Use Swedish subtitles if available.

Today our code looks something like this:

if (!captioningManager.isEnabled) {
  parametersBuilder.setPreferredTextLanguage("swe")
  parametersBuilder.setPreferredTextRoleFlags(C.ROLE_FLAG_SUBTITLE)
} else {
  parametersBuilder.setPreferredTextLanguageAndRoleFlagsToCaptioningManagerSettings(context)
}

But for a channel that has only captions available and system settings disabled usages of captions will still pick display captions if we call setPreferredTextLanguage("swe"). If we remove the call to setPreferredTextLanguage("swe") we have another case were an asset with multiple subtitles languages and the first track will be picked which is not always Swedish.

In a nutshell, is there any way to force usage of text role flag instead of setting the preferred?

[icbaker]

Just to clarify, are you distinguishing between 'captions' and 'subtitles'? These terms aren't clearly/distinctly defined in the DASH spec I'm looking at, so if you're ascribing particular meaning to them in your streams it might be useful to explicitly describe that meaning here (as it probably helps explain some of the context behind your request).

In a nutshell, is there any way to force usage of text role flag instead of setting the preferred?

Something like this?

class TrackSelectionParameters {
  class Builder {
    /**
      * Sets the role flags that a text track must have in order to be selected.
      */
    public void setRequiredTextRoleFlags(@C.RoleFlags int requiredTextRoleFlags) { ... }
  }
}

There's not currently a way to do that directly with the TrackSelectionParameters API - but it seems like a reasonable request. I'll have a go at adding it.

In the meantime, you can implement this more manually by calling TrackSelectionParameters.Builder.setOverrideForType from inside Player.Listener.onTracksChanged and ensuring you disable all the tracks you don't want to be selected.

[sobbe81]

Hi,

Yes, the DASH specification only lists caption and subtitle as possible values of the Role parameter. The terms captions and subtitles are probably something that is defined in how text is represented in video streaming / compression, i.e. outside any streaming format specification.

From: https://www.3playmedia.com/blog/closed-captioning-vs-subtitles/#:~:text=In%20the%20broadest%20sense%2C%20each,any%20language%20to%20watch%20video.

"Captions provide an accessible way for viewers who cannot hear audio to watch video. Subtitles provide an accessible way for speakers of any language to watch video"

So in our case

• We would like hearing impaired users be able to see captions as default if defined in phone settings (via CaptioningManager). These users will most likely want to have captions enabled for every playback. Users that are not hearing impaired will probably not want Swedish text on Swedish channels.
• We would like all users to see Swedish subtitles as default when Swedish subtitles are available. With the reasoning that most people in Sweden would like Swedish subtitles that would translate the dialog in the movie.

I think setRequiredTextRoleFlags would solve our problem. Would it still be possible to display other languages/roles to that the user can override our default choice?

[icbaker]

Users that are not hearing impaired will probably not want Swedish text on Swedish channels.

What about forced subtitles, i.e. Swedish subtitles when watching a mostly Swedish-audio content that has some parts of non-Swedish dialogue where you expect to always see subtitles matching the audio language.

If we implement setRequiredTextRoleFlags and you do setRequiredTextRoleFlags(ROLE_FLAG_CAPTION) then these forced subtitles won't be selected unless they are caption role - which by the definitions you've provided wouldn't be the case.

[sobbe81]

I see what you mean. What about something like this:

parametersBuilder.setRequiredTextLanguage("swe")
if (captioningManager.isEnabled) { parametersBuilder.setPreferredTextRoleFlags(ROLE_FLAG_CAPTION) } else { parametersBuilder.setRequiredTextRoleFlags(ROLE_FLAG_SUBTITLE) }

Always require Swedish text otherwise to text. Hearing impaired would prefer captions but subtitles would be used if captions isn't available. Non hearing impared would use subtitles if available otherwise no text.

[icbaker]

I discussed this a bit with the team today, and we have some further thoughts:

• We're reluctant to add the setRequiredTextRoleFlags() method that I proposed, and the setRequiredTextLanguage() method that you proposed because it will start to get very complicated to reason about how these interact with the existing setPreferredXXX() methods. For a taste of how complicated it already is to reason about these methods, see the discussion here of how they interact: https://github.com/google/ExoPlayer/pull/6158#discussion_r301955481
• We think we don't completely understand the context of your problem/question - so we have some follow-up questions (these all concern the case when caption manager is disabled - I think things generally work the way you want when caption manager is enabled?):
  1. Are you trying to show these subtitles 'by default' i.e. without the user explicitly toggling something in the UI?
  2. When you have content with both swedish subtitles and swedish captions, and you want to show the subtitles but not the captions, is the audio also in swedish (always/sometimes/never)?
  3. When you have content with only swedish captions and no swedish subtitles, and you therefore want to show no text on screen, is the audio always in swedish?
     • This last point is because we can't really understand why you would want to fallback from "show ideal swedish text" (subtitles) to "show no swedish text" when there's a "show less-than-ideal swedish text" (captions) option available - we would assume that if you're trying to show textual content to a user, it's generally better to show something textual than nothing. If you could clarify the reasoning here, it might help us to understand the bigger picture.

My general feeling is that there might be a different way to describe your problem, with a bit more information - and that might point towards a different solution e.g. something like (though this is a guess):

• If you want to always show swedish subtitles by default, without user input, and you control the media, you can just mark this swedish subtitle track as default in the media. Then with no special track selection parameters set this track should reliably be selected by default.

[sobbe81]

I agree that the apis becomes a bit too complex.

What we are trying to adress is what text track (if any) should be displayed to the user as default behaviour without user interaction. We would like all text track to be available for the user to choose while playing content.

We have linear channels where some programs has swedish audio and some that does not.

Some channels will have a subtitle AdaptationSet available in the manifest but there will only be actual text available for the programs that have non-swedish audio. We would like use swedish subtitles as default since many of our users would not understand the audio for program with non-swedish audio.

Then we have some channels that could have captions available in the manifest. In this case it's swedish text and swedish audio. This is intended for hearing impaired users.

What we would like to achieve is that:

Captions should be used as default if and only if enabled via CaptioningManager.

otherwise

Swedish subtitles should be used as default if available.

otherwise

No text as default

HLS specifies a DEFAULT attribute for text tracks but from what I know there is not such a thing in DASH. Or do you know how to specify that in a DASH Manifest?

[unext-wendong]

@icbaker Sorry for jumping in.

Regarding how to signal the default tracks in DASH manifests, I have a related question.

As @sobbe81 mentioned, in HLS, there is the DEFAULT attribute to signal the default tracks (the default track here means the track to select when there is no explicit user preference).

In case of DASH, from the following DASH spec, I believe it's the Role tag with the scheme urn:mpeg:dash:role:2011 and value main.

// From ISO/IEC 23009-1 Fifth edition 2022-08, section "5.8.5.5 DASH role scheme":
Main media component(s) which is/are intended for presentation if no other information is provided.

However, it seems the current implementation doesn't take it into the consideration during track selection. And from the test, it doesn't select any text tracks by default when the Captions system setting is OFF even though there are text tracks marked with the main role.

Adding a "switch" case here for "main" to return C.SELECTION_FLAG_DEFAULT might be enough to fix it: https://github.com/androidx/media/blob/release/libraries/exoplayer_dash/src/main/java/androidx/media3/exoplayer/dash/manifest/DashManifestParser.java#L1530

Would you take a look?

Btw, confirmed that shaka-player is by default selecting the "main" tracks.