Bind subtitle choices between AirPlay sender and receiver

[defagos]

When using AirPlay, choosing a language for subtitles either on a casting device or on an Apple TV receiver should update the preferred language on the casting device. Currently, changing the subtitles on the Apple TV receiver has no impact on the casting device.

Issue type

Incorrect behavior

Environment information

• Library version: 3.0.0, but probably all versions since 2.5.6 at least.
• iOS version: Any
• Device: Physical

Reproducibility

Always reproducible

Steps to reproduce

Perform the following steps with SRG Media Player demo, first with the system player, then with the advanced one. Compare the results:

1. Play the Apple Advanced stream example.
2. Set subtitles to off on the mobile device.
3. Enable AirPlay.
4. Change subtitles to French with the Apple TV remote.
5. Close the player.
6. Disable AirPlay from the control center.
7. Open the same example again.

With the system player French subtitles will be automatically selected when reopening the player. With the advanced one this is not the case.

[defagos]

I also spotted additional issues:

• When casting with AirPlay, the Automatic setting is disabled in the tracks popover but can still be tapped.
• When changing the language on the Apple TV receiver with the tracks popover opened on the casting device, the language is not updated accordingly.
• While using AirPlay, changing subtitles very fast leads to an incorrect result (probably because we force an table update immediately when selecting the cell).

These issues are nowhere to be found in 2.5.6, the version which introduced most improvements related to subtitles management. I think they probably appeared due to the KV-observation changes introduced in 3.0.0.

The main problem discussed in this issue, though, still existed in 2.5.6.

[defagos]

Note that, though we should reflect Apple TV language choice changes on the casting device, the opposite is not possible, i.e. it is not possible to change the Apple TV preferred language based on the last choice on the casting device:

1. Open SRG Media Player demo and play the Apple Advanced stream with the system player. Set subtitles to Off.
2. Enable AirPlay.
3. Change the language to French on the casting device.
4. Close the player, disable AirPlay.
5. Open the Letterbox demo Apple TV app, play a 19h30. Subtitles are not restored to French.

The reason is that we cannot call MACaptionAppearanceAddSelectedLanguages on the Apple TV receiver. This method is correctly working on Apple TV, though, as can be seen with the following steps:

1. Open SRG Media Player tvOS demo and play Apple Advanced stream. Set subtitles to French.
2. Then open Letterbox tvOS demo. Play a 19h30. French subtitles are enabled automatically.

[defagos]

For the easy part, I fixed the incorrectly selectable Automatic item (the didSelect method is called in all cases, even if the cell as no selection style set).

As always with subtitles, the problem is quite more complex than it initially seems to be, so the rest of the task required more thought.

First, on a device, MediaAccessibility local settings (the previously selected subtitle option) are automatically applied when playing a media. This is not true, though, when starting playback while external playback is already enabled. In such cases, the last language used on the Apple TV receiver is used. This language is in general not the same as on the mobile device and will override it (on Apple TV, the same MediaAccessibility framework for tvOS is used for saving preferences, as on iOS). This is the issue reported by our Play SRF user.

To fix this issue, we must not rely on the standard AVPlayer mechanism, but read MediaAccessibility settings and apply them manually when a AirPlay session is already enabled. This could be done only in this case, but I chose to do it in all cases (i.e. also for local playback). This guarantees we have the behavior we want in all cases, even with future iOS versions (even if it is likely this will not change).

I also had some doubts about what to do if the language is changed on an Apple TV while using AirPlay:

• Should we treat the receiver and sender as separate devices with unrelated media accessibility settings?
• Should we mirror back this change to the device?

The first option is appealing since we cannot alter the subtitles settings on an Apple TV when the user changes subtitles on the sender casting content via AirPlay (the system does not save these changes, and we have no way to call the MediaAccessibility tvOS API from the sender through AirPlay).

If we see the sender app as a kind of remote (like the Apple TV remote), though, the second option is more appealing. This introduces some asymmetry (receiver subtitle changes can update sender settings, but not conversely), but I finally think this helps us provide a better user experience altogether. Moreover, control center subtitle change implementation automatically performs the same settings update on the sender, without the need to call the MediaAccessibility iOS API (the change is induced by the update on the receiver). This means that no Letterbox update is required.

These changes have been made on the feature/airplay-subtitles-selection-fixes branch. I intentionally kept the intermediate commits reflecting these two possibilities. We can discuss whether the choice I finally made is better or revert to an older version.

Finally, I do not recommend fixing the issue when changing subtitles over AirPlay fast. This is a minor selection issue. If we can find a simple fix this is good, but I strongly recommend not triggering periodic time observers faster just for the sake of fixing this issue.

[defagos]

In some cases, we could have no selected option. One use case was when switching to Off or Automatic on an Apple TV receiver. Another case was if the previously selected option (e.g. Japanese) was not available for another media being played.

In such cases, I propose we display the local setting to the user, grayed out, and explicitly set to be unavailable. This way the user will be informed that the device looked for some subtitles, but could not find them. Once the user selects a valid language this grayed out entry will be removed.

[defagos]

After more tests, I made some more changes and reverted other ones:

• Subtitle changes are not mirrored back from the AirPlay receiver to the device. After giving this feature even more thought, I think it is probably better to treat sender and receiver separately. We namely have a media configuration block which could be reloaded during AirPlay playback, leading to a change which we cannot distinguish from a change made by the user with the Apple TV remote. What remains to be seen is the user experience we now obtain when toggling subtitles from the control center (local settings update was free with the previous implementation). My first feeling is that the control center should be seen as a kind of Apple TV remote (and thus not update device subtitle settings via MediaAccessibility), but I first need to check the result in Letterbox to see whether this makes sense.
• I further improved the initial media configuration so that it matches the system behavior as close as possible (this is the case when appliesMediaSelectionCriteriaAutomatically is enabled, which is the default). In particular, SDH tracks are now loaded if available and if the corresponding Accessibility setting is enabled. Having written the whole implementation, I also attempted to disable appliesMediaSelectionCriteriaAutomatically (since we override this behavior anyway), but we lose automatic AD audio track selection if the corresponding Accessibility setting is enabled. Since there is no API to know whether this setting is enabled or not AFAIK, I decided to keep appliesMediaSelectionCriteriaAutomatically default value, and the equivalent implementation override.
• I further improved subtitle information display in the tracks popover. During my tests, I played with the media configuration block, and remember our original implementation did not address changes made using this block. Our original implementation reflects the setting saved with MediaAccessibility, not really the subtitles being used. This is quite confusing for the user (unforced subtitles might be displayed while Off is ticked), so I decided to improve the display as follows:
  • If the MA setting is Off (i.e. only forced subtitles), then if unforced subtitles are set we check the corresponding subtitle entry, and leave Off unchecked. This way what the popover displays is what the user actually sees, not the MA setting. The MA configuration stays as is, except of course if the user taps on an item in the popover, in which case it will be updated as before.
  • If the MA setting is Automatic, we display any unforced subtitle track language as detail text in the same cell for information purposes. We cannot check the cell corresponding to this language itself, as we have no way of knowing whether the unforced track has been selected because the media configuration block forced it, or because automatic selection picked it. Having two checkboxes (one on Automatic, the other on the language) would also be misleading IMHO. So I propose this compromise. Of course, another possibility is simply to select Automatic without additional information, which would also be valid from a user perspective.
  • If the MA setting is Off, we check the language currently used, whether it is the one recently used, one forced by the media configuration block, or an unsupported language (in which case an additional disabled cell is introduced, see above).

[defagos]

I finally understood that the AVPlayerViewController automatic subtitle selection is buggy. It is namely obvious that tapping on Automatic sets the corresponding MediaAccessibility mode, kMACaptionAppearanceDisplayTypeAutomatic, as the behavior is persisted across apps and players.

The documentation of this setting is explicit: When automatic is used, and if the audio language does not match the one of the device, subtitles matching the device language will be used if available. As can be easily seen with SRG Media Player demo and the Advanced Apple example, this does not work correctly. You also have to enable CC & SDH in the Accessibility settings for the test below (the unforced subtitles available for this stream are all SDH, and automatic AVPlayerViewController selection does not find them if the setting is not enabled, which could be seen as a bug, as the kMACaptionAppearanceDisplayTypeAutomatic documentation never mentions this should be the case):

• Set the device language to French.
• Open the system player demo and play the Advanced Apple example, go to the subtitles and set the behavior to Automatic.
• Close the player.
• Open the same example again. The audio does not match the one of the device, French unforced SDH subtitles are available, we expect them to be displayed. This is not the case, though.
• Go to the subtitle and audio settings and tap on Automatic . This time Automatic is disabled and French subtitles are enabled instead, which is the correct behavior.

For consistency, we also expect automatic mode to update subtitles later if the audio track is changed. This expectation can be tested with the Gens d’Hiver example, which has French audio and subtitles, and other audio tracks:

• Open the Gens d’Hiver example with the system player.
• Open subtitles and audio settings, choose Italian, tap on Automatic. This will display French subtitles (tapping is required as the initial automatic choice is not made, as discussed above).
• Now change audio to French. We would expect subtitles to be hidden automatically, but this is not the case.

I therefore completely rewrote our existing implementation to implement proper automatic selection. The resulting behavior is different from the buggy AVPlayerViewController one, but is clear and consistent:

• When starting playback, subtitles are displayed according to kMACaptionAppearanceDisplayTypeAutomatic documentation when automatic mode is enabled.
• Automatic selection prefers SDH subtitles if the corresponding Accessibility setting is enabled, otherwise it still considers standard unforced subtitles.
• I finally propose to display the subtitle selected as double selection. This clearly informs the user about which subtitle is used, if any, and makes overrides to automatic selection (via a media configuration block) explicit as well. The user can then tap on Automatic to revert to standard behavior, at which point the language change is clearly visible.
• When audio tracks are changed, automatic mode updates subtitles accordingly, disabling or enabling them depending on whether the audio language matches the device language or not.

In rewriting the implementation, I also magically fixed the selection issue appearing when switching between subtitles very fast while AirPlay is active.

Finally, I improved the existing tracks popover implementation, which could not automatically update if the media is changed for some reason while the popover is displayed.

[defagos]

It would probably be helpful to report these issues to Apple (maybe rather reporting them as Safari issues, but also mentioning AVPlayerViewController). Apparently, these are not regressions (the bugs existed in prior iOS versions), but they might consider fixing them.

I should also test if these issues affect tvOS as well (very likely). The AVPlayerItem automatic media option selection API might also be buggy, as is the automatic selection behavior when starting playback while an AirPlay session has already been established.

If these bugs were fixed, our implementation could be made simpler (the logic implementing these expected behaviors could be simply dropped).

[defagos]

I also improved instructions displayed on the popover. Beware of the difference between subtitles and CC.

In French, CC is STC (Sous-titres codés).

[defagos]

Here is a list of all possible configurations users will see when opening the tracks popover or changing parameters. Unlike what we had with our previous implementation, and unlike what happens with AVPlayerViewController implementation, an entry is always selected, and proper context information is provided to the user about what she might have previously selected and what the Automatic option does.

No subtitles

The user has explicitly disabled subtitles. Only forced subtitles can be displayed.

Automatic mode

When using automatic mode, subtitles may not be necessary if the audio track language matches the one of the device (here French audio on a device in French):

If this is not the case, and if subtitles matching the device language are found, they will be automatically enabled (here Italian audio on a device in French):

Explicit subtitle choice

If the user has explicitly selected a language, the corresponding entry is selected:

If the user selected a language for another content, and the language is not available for the current content (maybe in another app like Safari), the setting is Off and the most recent language is displayed grayed out for information:

The user is free to choose another setup, which makes the grayed out row then disappear.

[defagos]

A final comment about the tvOS implementation. The initial subtitle choice is the same as for iOS. Since we are using AVPlayerViewController, though, we will not benefit from the same improved user experience as on iOS, but merely stick with the system one. That means some improved behaviors (e.g. automatic subtitle changes when the audio track is changed) are not available for this version.

It would probably be possible to hook into the system implementation. I would rather avoid implementing such user-driven changes by monitoring track changes at the controller level. These changes would namely be difficult to distinguish from non user-driven changes. For this reason, I propose we do nothing special for the tvOS implementation at the moment.

[pyby]

feature/airplay-subtitles-selection-fixes merged.

[pyby]

We found a strange behaviours, both on SRGMediaPlayer avec System AVPlayer:

In the Demo application:

1. Connect your iOS device to AirPlay
2. Play "Bonjour la suisse" with the Advanced custom" player. It's playing on the Apple TV.
3. Open the alternate tracks view on the iOS device. Check that selected audio track and selected subtitles track are corrects. (in example, French audio and subtitles OFF).
4. With the Apple TV remote, change audio track. (in example, switch to German audio: the sound is now in German and the alternate tracks view on the iOS device is updated).
5. On the device, in the already opened change subtitles selection (in example, switch to Italian subtitles: the sound is now in German).

On point 5: the subtitles track is updated correctly, but the audio track roll backs to the first audio track selected on the iOS device (in example, audio track switch to French and subtitles to Italian).

We have this issue on both AVPlayer and SRGMediaPlayer. We can't fix it easily. @defagos and me suggest to report this bug to Apple.

[defagos]

After more tests with production streams, I found an erratic behavior which was leading to subtitles sometimes incorrectly not automatically selected, even if the audio did not match the system language but subtitles in the system language were found instead.

This was related to audio option selection sometimes not immediately reflected by -srgmediaplayer_selectedMediaOptionInMediaSelectionGroup: when looking for subtitles. I was namely using this method, as I thought there was no way to implement audio track selection automatic selection manually while still taking into account Audio Description preference. I was wrong, there is such an API in MediaAccessibility, namely MAAudibleMediaCopyPreferredCharacteristics.

I therefore could implement a proper automatic audio selection algorithm, mimicking the one of the system, and returning the choice made so that it can be immediately reused as input for subtitle selection. This way unreliable results due to the use of -srgmediaplayer_selectedMediaOptionInMediaSelectionGroup:, which would not always immediately reflect the applied value, are eliminated.

I also improved the audio track resolution approach in comparison to AVPlayerViewController, so that the application language is taken into account as well. For our French-only app, for example, this means that the French track is selected by default, even if an English track is available and the device is set with English as only preferred language.

This fix is available on the feature/media-configuration-improvements branch.

[defagos]

I also had to update the audio and subtitle choice API to better separate both.

It was namely a problem to decide what to do if the user was using a mediaConfigurationBlock, previously used for both audio and subtitle language choice (as well as subtitle styling). For example, what should have been done if the user decided to select a different audio only? How would subtitles behave in this case? Our implementation could not guess whether the user had changed subtitles as well or not, and the only choice we could do in this case was to start without subtitles before customization was applied. This means that if the user was somehow only changing the starting audio, we could not provide an automatic subtitle choice.

Moreover, having to reload the whole configuration for subtitle styling was clearly wrong. Especially with a single block disabling subtitles if implemented, this would have meant that clients which wanted to customize the subtitle color would have lost the automatic selection behavior.

To solve these issues, we now have two configuration blocks, one for audio, the other for subtitles. This provides support for all possible implementation needs. Moreover, styling is now simply made via a dedicated controller property.

I also removed the configuration reload methods having a block parameter. It did not make sense to have such a method with two blocks, and removing it for medias meant we should remove it for player configuration as well.

This is a breaking change, but for the best. This will make #82 even easier to implement afterwards.

[defagos]

I made further improvements to the current behavior:

• Do not display STC without language unnecessarily. This avoids having a useless / annoying / misleading tracks button for many streams (e.g. Tageschau or Apple basic examples) and avoids subtitle choices which, from a user perspective, do nothing.
• When subtitles for a given language are automatically resolved, take into account the SDH / CC accessibility preference consistently.

[defagos]

Here are a few examples of how audio tracks and subtitles (in automatic mode) are now selected, after all the changes discussed in this issue have been made.

Video content is assumed to be available with the following options:

Audio: French, French AD, English Subtitles: French, French CC, English

Use case 1

Setup:

• App languages: French
• System preferred languages: French
• Prefers AD: Off
• Prefers SDH & CC: Off

Result:

• Audio: French
• Subtitles: None

Note that if the user switches the audio to English, French subtitles are automatically enabled.

Use case 2

Setup:

• App languages: French
• System preferred languages: French
• Prefers AD: On
• Prefers SDH & CC: Off

Result:

• Audio: French AD
• Subtitles: None

Note that if the user switches the audio to English, French subtitles are automatically enabled.

Use case 3

Setup:

• App languages: English
• System preferred languages: French
• Prefers AD: Off
• Prefers SDH & CC: Off

Result:

• Audio: English
• Subtitles: None

Use case 4

Setup:

• App languages: English
• System preferred languages: French
• Prefers AD: On
• Prefers SDH & CC: Off

Result:

• Audio: French AD
• Subtitles: None

Since the user has a system in French, we assume it can understand French, and therefore AD takes precedence over English (the app language, in general the default selection).

Use case 5

Setup:

• App languages: French
• System preferred languages: French
• Prefers AD: Off
• Prefers SDH & CC: On

Result:

• Audio: French
• Subtitles: French CC