Turn captions on by default in iOS

[elynema]

Description

As of #5084, captions could be enabled on iOS devices in full-screen player by selection English as the subtitle option. We would like the captions to be turned on by default if possible, and also to work if Auto is selected under the subtitle mens rather than Englishe.

Definition of Done

• [ ] On iOS device, once playback begins in full-screen mode, captions display automatically if available
• [ ] On iOS device, once playback begins in full-screen mode, captions can be turned off using the subtitle menu
• [ ] On iOS device, once playback begins in full-screen mode, captions can be re-enabled by selecting "English" or "Auto" options.
• [ ] Captions work as normal on browsers (generally toggle on/off with CC icon; on by default)

[masaball]

I have looked into this for a bit and done some testing. For reference, here is the RFC spec for HLS streaming, specifically the section dealing with relating other Renditions of a media stream, which is what subtitles/captions are classified as (4.3.4.1 EXT-X-MEDIA). There is good info on the whole setup there but what is relevant is on page 26. The attributes we care about for captions being on by default are:

• Default: "If the value is YES, then the client SHOULD play this Rendition of the content in the absence of information from the user indicating a different choice."
• Autoselect: "If the value is YES, then the client MAY choose to play this Rendition in the absence of explicit user preference because it matches the current playback environment, such as chosen system language.
• Forced: "A value of YES indicates that the Rendition contains content that is considered essential to play. When selecting a FORCED Rendition, a client SHOULD choose the one that best matches the current playback environment (e.g., language).

I have tested all 3 in different combinations but have had no luck getting them to be on by default when using BrowserStack.

Additionally, I encountered behavior with the standard Avalon player where having DEFAULT enabled caused the captions to render as part of the video but they could not be turned on or off. The Avalon player's CC button would turn on/off our sideloaded captions but had no effect on the HLS captions, so if the CC button was on there would be 2 sets of captions displayed. So, if DEFAULT were to end up being required for iOS default on captions it might require us to rework how the native Avalon player handles the captioning.

All of that being said, I eventually found this Apple support page that indicates it may be up to each individual how the Apple video player defaults. I don't think there is a way to test this in browserstack though, as you only have access to one of the browsers and cannot mess with the device's settings. If someone on the team has an iPhone or iPad and the IU VPN setup on it, we could test on Avalon-Dev if turning on the iOS device's accessibility option gets the captions to turn on in the player as the AUTO option. I don't have any iOS products so one of the other team members would need to test if it works with the configuration currently on Dev.

[Dananji]

@masaball I tested this on an iPhone (12 Pro, iOS 16.31) by changing the device's accessibility settings;

• With device's CC settings turned on: subtitles are displayed but the selected option is not AUTO; And when I exit the full-screen player, 2 sets of subtitles are visible in the player (I'm guessing one from the Apple's player and one from Avalon's player?);

• With device's CC settings turned off: subtitles are not displayed but the selected option is AUTO in Apple's native player; And when I exit from the full-screen player subtitles are displayed in player;

[masaball]

Thanks for testing this Dananji. Note: Dananji said during standup that the device CC settings were defaulted to on.

• As long as the captions are coming on by default when the device CC settings are on, I don't think it really matters one way or the other if they render when AUTO is selected.
• I've seen discussion that setting FORCED=YES is what gets caption to render when auto is selected but that did not seem to work when I tested with Avalon-dev through BrowserStack. So, I am not really sure how to get our captions to render with Auto. Maybe if the device settings are on and the FORCED attribute is enabled?
• I think Dananji is correct that the 2 sets of captions rendered on Avalon player after leaving the AVplayer is that both our sideloaded captions and the HLS captions are rendering.
• In the first screenshot, the duplication of caption choice bug has shown up again. I did a quick check on BrowserStack and I think this may be a bug with iOS 16 specifically because I did not encounter it in iOS 15. The behavior can be duplicated by turning captions on in the AVplayer, navigating to a new page then returning to the original video. My inclination is still that it is a weird caching error of some kind.

[elynema]

My testing in ios 15.7 on iPhone 13 mini

I do not have any accessibility settings turned on for captioning:

On avalon-staging, this is what I see when the video loads (captions are displaying). Note that captions turn off I select auto option and back on if I select either English option.

On mco-staging, this is what I see when the video loads (captions do not display; I have to select CC and then English before they display):