avalonmediasystem / avalon

Avalon Media System – Samvera Application
http://www.avalonmediasystem.org/
Apache License 2.0
93 stars 51 forks source link

Add captions to HLS manifest #5084

Closed masaball closed 1 year ago

masaball commented 1 year ago

Description

Closed captions for iOS have to be transcoded into the video. We can accomplish this by inserting caption information into the main HLS manifest which then calls a caption manifest.

Build off the investigation and proof of concept from #4951.

To Do

Definition of Done

elynema commented 1 year ago

If captions are included in both IIIF manifest and HLS manifest, could this cause a conflict with other IIIF players? Community is interested in segmented captions with HLS streaming files (request for cookbook recipe), so may be that nobody in the community is doing this yet. Might be good to test against Clover, as it's the only player rendering captions according to the viewer matrix.

elynema commented 1 year ago

Mason did some testing with Clover where captions are sent with both IIIF manifest and HLS manifest, which does produce inconsistent behavior based on how you selected CC in the video (sometimes none, sometimes 2 sets of captions, etc.). Currently, Avalon IIIF manifests do not provide transcripts. If we provide only HLS manifests to Clover (no IIIF manifest), the captions are an option in the player, but don't show up as a transcript.

elynema commented 1 year ago

Decision: address the potential conflict between IIIF manifest captions and HLS manifest captions when we actually implement transcripts/captions in our IIIF manifests.

elynema commented 1 year ago

@masaball In demo, English option was showing up twice in player under CC. Check this before going to QA.

elynema commented 1 year ago

@masaball I was doing some QA on this, and I seem to be finding the duplicate "English" problem cropping back up.

On all iOS browsers (Chrome, Firefox, and Safari), the captions are actually on by default, which is great. However, when I look at the subtitles menu, English is showing up twice. Turning captions off is working; I have to select either of the "English" options to turn them back on ("Auto" doesn't work). The latter is probably ok, but the duplicated "English" option is weird.

I thought we had given up on captions auto-playing on iOS for now, so this behavior is unexpected.

Note that I did encounter a few oddities where coming back to a video in Chrome, it seemed to work like it does in mco-staging (captions not on by default, "English" option only shows once). But on page reload it seems to behave in a default manner again.

I think this is the example: https://avalon-staging.dlib.indiana.edu/media_objects/g158bh28p

EE8B9FF3-006A-417B-95C6-EE2DD9507639

joncameron commented 1 year ago

Emily will retest on a phone; some amount of inconsistency is still here. We can refine this, but since it's functional we'll proceed with this setup so users can at least get the captions delivered.

joncameron commented 1 year ago

Future comment to go into the issue on autoplay captions.

elynema commented 1 year ago

Note that currently, on my personal device, the auto-play is more functional the mco-staging, so worth it to keep 7.6 functionality even if it looks a little wonky. On mco-staging, I am finding that I have to select CC and then English in order for captions to display, which is much more difficult.