Add Forced Subtitles Support

[Coralielm]

Description

As some content could have Forced Subtitles in their manifest, actually Dash.js player (4.5.2) display them in subtitles selector.

It's an issue in accessibility point of view: "Forced subtitles are a type of burned-in-text used to provide clarification on the plot or details of a video. Also known as forced narrative subtitles or forced captions, they provide the viewer with further information such as a translation of speech or signs or explanations regarding symbols, flags, slang words, texted graphics, etc.  In essence, we can think of the forced subtitles as the viewer’s assistant, offering them hints about the subtext of the production or clarifying what is going on on-screen." Source: https://www.rev.com/blog/subtitles-blog/what-are-forced-subtitles

Standards

DASH IF IOP Standard:

• https://github.com/Dash-Industry-Forum/DASH-IF-IOP/issues/165

Role@value: forced-subtitle Description: Textual information meant for display when no other text representation is selected. It is used to clarify dialogue, alternate languages, texted graphics or location/person IDs that are not otherwise covered in the dubbed/localized audio. Applicable Media Type(s) (Informative): text

HLS Standard:

• https://www.mux.com/blog/subtitles-captions-webvtt-hls-and-those-magic-flags
• https://datatracker.ietf.org/doc/html/draft-pantos-http-live-streaming-23

FORCED
      The value is an enumerated-string; valid strings are YES and NO.
      This attribute is OPTIONAL.  Its absence indicates an implicit
      value of NO.  The FORCED attribute MUST NOT be present unless the
      TYPE is SUBTITLES.

      A value of YES indicates that the Rendition contains content which
      is considered essential to play.  When selecting a FORCED
      Rendition, a client SHOULD choose the one that best matches the
      current playback environment (e.g. language).

      A value of NO indicates that the Rendition contains content which
      is intended to be played in response to explicit user request.

Behavior proposal expected for the player

When a forced-subtitle is present in manifest, the player should display it if no other subtitles is selected. Also this subtitle is not displayed in subtitles selector. A forced-subtitle refers to the same audio language. For ie. if an english audio is selected then the forced-subtitle in english should be displayed.

Sample Manifest

https://arteamd1.akamaized.net/GPU/034000/034700/034755-230-A/221125154117/034755-230-A_8_DA_v20221125.mpd

<AdaptationSet mimeType="application/mp4" lang="deu">
      <Label>Tedesco (forzato)</Label>
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="forced-subtitle" />
      <Representation id="14" bandwidth="856" codecs="stpp">
        <BaseURL>medias/034755-230-A_st_VA-ALL.mp4</BaseURL>
        <SegmentBase indexRange="961-2192">
          <Initialization range="0-812" />
        </SegmentBase>
      </Representation>
</AdaptationSet>

[dsilhavy]

To confirm after a discussion in an email thread, this is how Netflix is supporting forced subtitles:

They are not shown to the user/app in the selection dropdown They are automatically activated once there is no other subtitle selected The normal “subtitle” tracks contain the information from the “forced-subtitle” tracks. Consequently, we don’t need to handle two separate tracks simultaneously.

[nigelmegitt]

The Netflix model described is one where there is some subtitle track playing and everything in it is displayed; the player chooses either the "forced-subtitle" track or the "everything" track.

An alternative model to this is supported in IMSC:

• the subtitle document contains both "forced" and "non-forced" subtitles, each type identified by the computed value of itts:forcedDisplay.
• the player passes a displayForcedOnlyMode flag to the subtitle renderer to tell it whether to show only those subtitles where itts:forcedDisplay="true" or all the subtitles.

Ideally both modes would be supported.

Either way, the presence of a subtitle file that contains forced subtitles in the same language as the user's chosen audio track needs to be identified so that the subtitle file can be loaded and played.

[mikedo]

Although we contemplated this in TTWG, it is more complex. Has anyone implemented this? If not, I would encourage just the simpler implementation for now.

[nigelmegitt]

The displayForcedOnly method is implemented in imscJS, that is already integrated into dash.js - it's done by passing an appropriate value of displayForcedOnlyMode parameter into the imscHTML.render() function.

The complexity is in the signalling in the MPD, deriving what subtitle track should be active, and passing in an appropriate value to imscJS. I suspect that most of that complexity is common between both methods.

[hd51515]

Looking for an example MPD that can show subtitles (for my understanding on how the subtitle code works). The MPD here gives subtitle options (Fr, De) but does not display anything when selected. There are also multiple audio options and it is not clear how to make the player select a specific language audio. https://arteamd1.akamaized.net/GPU/034000/034700/034755-230-A/221125154117/034755-230-A_8_DA_v20221125.mpd

[dsilhavy]

@hd51515 Thank you for looking into this. There is a single caption for the forced german and french track from 0 to 3:

        <tt:p style="textAlignCenter" end="00:00:03.000" begin="00:00:00.440" region="bottomAligned" xml:id="sub2">
            <tt:span style="YellowOnBlack doubleHeight">BISHER</tt:span>
            <tt:br/>
        </tt:p>

But maybe @Coralielm has an MPD with more entries?

For track selection you can check: https://reference.dashif.org/dash.js/nightly/samples/multi-audio/multi-audio.html and follow the code flow starting from player.setCurrentTrack(self.player.getTracksFor(item.mediaType)[item.index]);

[hd51515]

using this HTML for reference https://reference.dashif.org/dash.js/nightly/samples/captioning/multi-track-captions.html

[Coralielm]

HI @dsilhavy isn't my example here sufficient? https://arteamd1.akamaized.net/GPU/034000/034700/034755-230-A/221125154117/034755-230-A_8_DA_v20221125.mpd

What do you need exactly?

[razvanmarianagache]

@ShikiSeiren Would you happen to have any idea or progress about the forced-subtitle ?

I was looking into this as well and noticed an odd behavior.

The forced subtitle in our mpd's are marked as roles: ['main', 'forced-subtitle'] or roles: ['forced-subtitle']

Although I am setting the correct text track id and dash player seemingly correctly applies the new text track id this is still not working with the latest version 4.7.3

I clearly see this being applied in dashjs_text_settings from local storage but it looks like the previously selected subtitle ( with the same lang ) will get activated instead...

I'm wondering if cancelling out the caching mechanism of the .vtt files that are being loaded would fix this.

[dsilhavy]

@razvanmarianagache What exactly are you implementing right now? Is it support for forced-subtitles in version 4.7.3? Or are you trying to set a specific subtitle via your application using the dash.js API? Moreover, can you share the MPD you are using in your tests and the potentially relevant code snipped of your application that does the track switch.

[razvanmarianagache]

Hi @dsilhavy I have forked dash and I was trying to figure it out myself ( I had no knowledge of the codebase at all or architecture prior to this ).

I am setting the index of the subtitle and this process seems to be working perfectly fine. I also see the correct .vtt file being downloaded. Dash applies it correctly it seems as I see from some logs; but the display on screen is wrong so I'm thinking maybe the old buffer for texts is not being properly cleared or somehow it gets combined with the one for forced-subtitles...

forced-subtitle-mpd-test.zip

[dsilhavy]

Did you make any additional changes to dash.js itself in your dash.js fork? The code snippet above looks like application logic around dash.js which does not require any changes to the existing dash.js code base.

Make sure to use the dash.js API for switching between texttracks (player.setTextTrack) and do not operate directly on the video element. If you see cues being rendered that do not belong to the active texttrack I suggest looking into the video element directly since we use native functionality for WebVTT cues by default:

videoElement.textTracks --> find the active one and check the cues attribute.

In case you are using the custom rendering (https://reference.dashif.org/dash.js/nightly/samples/captioning/vttjs.html) I suggest following the logic in the dash.js player and check if old cues are not removed correctly.

In both cases (native and custom rendering) you can test in Desktop browsers and work with breakpoints which makes debugging easier.

[razvanmarianagache]

Hi @dsilhavy Following the cues, proved to be productive and I managed to have this working for me. Although I feel this is a hack to cover a hidden issue someplace else but after spending so much time on it, I'm inclined to embrace the 'if it's works then don't touch it strategy'.

When I was looking into this I noticed the cues not being ordered properly when switching from forced so some other caption or subtitle.

Right now I'm loading the 'dash.all.min.js' generated by the local dash fork I got. It doesn't seem to have any memory leaks ( so far... ).

Changes that I did to fix this: 'src/streaming/text/TextTracks.js'

[stschr]

As commented to https://github.com/Dash-Industry-Forum/dash.js/pull/4545 , just one thought on

For that reason, the language of the current audio track is compared to the available forced-subtitle language options.

Per NorDig IRD spec, clause 16.2.1, every TV is mandated to not only store audio language preferences, but also a "primary subtitle language". Likewise in ATSC3, where the "Query Language Preference API" returns preferredCaptionSubtitleLang (clause 9.2.4).

Having this in mind, wouldn't it make sense to provide subtitle language as an option in Settings.js and just fallback to current audio language if no setting is provided?

Of course, this would not only impact "forces subtitles, but also regular subtitle selection.

[dsilhavy]

As commented to #4545 , just one thought on

For that reason, the language of the current audio track is compared to the available forced-subtitle language options.

Per NorDig IRD spec, clause 16.2.1, every TV is mandated to not only store audio language preferences, but also a "primary subtitle language". Likewise in ATSC3, where the "Query Language Preference API" returns preferredCaptionSubtitleLang (clause 9.2.4).

Having this in mind, wouldn't it make sense to provide subtitle language as an option in Settings.js and just fallback to current audio language if no setting is provided?

Of course, this would not only impact "forces subtitles, but also regular subtitle selection.

Thanks @stschr for the input. To me this sounds like a change that is not directly related to what I am doing here. I like the idea of using the system defaults as an input for the initial language selection at start. As you are aware, we are also using the local browser storage today to save the last settings.

From my perspective the selected forced-subtitle should match the current selected langauge even if it does not match the default "primary subtitle language". As an example:

1. The default subtitle language is german
2. The user selects the english audio track
3. No subtitles are enabled, dash.js will now enable the english forced subtitles. As a user I would find it irritating if another language is selected here.

But I am open for feedback and opinions. Maybe I misunderstood the concrete use case you had in mind.

[dsilhavy]

@Coralielm , all I have merged the changes of #4545 into development. It would be nice if everyone who is interested and has test content checks the current implementation and provides feedback.

I leave this issue open for feedback. I will also create a new issue for the input that @nigelmegitt gave earlier regarding the IMSC model

[dsilhavy]

First tests looked good, closing as completed for now