How to add audio descriptions to player?

[KianBadie]

Description

I've been trying to search online for examples/tutorials/documentation on adding audio descriptions to videojs in order to meet wcag 1.2.1. However, I haven't had much luck! Even trying to just show the audio descriptions button has been a challenging thing for me to figure out. This code snippet doesn't result in the button showing, and I'm not sure what else would make it show (ellipses added for brevity):

let videoPlayer = videojs(root, {
        controls: true,
        controlBar: {
            ...
            descriptionsButton: true,
            ...
        }
});

Would anyone be able to offer some guidance on adding audio descriptions or have some resources to utilize?

[OwenEdwards]

@KianBadie there are two ways to add audio description using video.js, and there are examples on the videojs.com website - looking at the HTML and source files used in each of those examples may provide what you need:

1. Using a separate audio track that includes a mix of the main audio track and the audio description, and including that in an HLS format source so that users can select either the normal audio track or the one with audio description from the "Audio Track" menu, where the audio description track is marked with the "AD" icon. See: https://videojs.com/advanced/?video=elephantsdream

2. Using a text track with audio descriptions, so that the audio descriptions are essentially like captions, but when they are displayed by the player they are tagged with aria-live so that, if the user is using a screen reader, the screen reader announces each audio description when it is displayed. In this example, the user must turn captions off, and then the "audio description" button is enabled. See: https://videojs.com/advanced/?video=elephantsdreammp4

The second method has some significant limitations, and although it is supported by the HTML5 and WebVTT specifications, it is not clear that it satisfies WCAG 1.2.3 or 1.2.5 (see Advisory Technique H96: Using the track element to provide audio descriptions. The first method is capable of complying with Sufficient Technique G78: Providing a second, user-selectable, audio track that includes audio descriptions, but whether it does actually satisfy WCAG 1.2.3 and/or 1.2.5 depends on other factors, including the quality and completeness of the audio description that is added.

I need to note this disclaimer, though - broadly speaking, video.js aims to support WCAG 2.0 Levels A and AA, but cannot provide any guarantee that using video.js in a website will be compliant with those WCAG guidelines, nor with ADA or Section 508. See video.js Wiki: Accessibility (a11y) is video.js accessible ADA compliant WCAG compliant?

[KianBadie]

@OwenEdwards thank you for the help and the examples! I will look more into the examples on the videojs website that you provided. I have a couple questions from your examples:

Does the audio file format need to be HLS? Can it not be mp4? And, out of curiosity, why wasn't the dedicated audio tracks button used in the first example?

Also, thank you for the disclaimer and explanation of how each technique relates to conformance. It is very thorough and helpful! I will keep the disclaimer in mind.

Edit: Also, I am having a bit of trouble trying to find where the audio descriptions are added in the advanced examples. I have tried reading through some of the files in the advanced examples folder for the site but have not found audio description related code yet. Would you happen to know if I am looking in the right folder?

[misteroneill]

Does the audio file format need to be HLS? Can it not be mp4? And, out of curiosity, why wasn't the dedicated audio tracks button used in the first example? Essentially, yes. It could also be DASH, but the main point is that it would be an alternate audio track within a single stream.

As Owen said, this would appear in the Audio Tracks menu not the Audio Descriptions menu. Audio Descriptions are a text-based format (VTT) like all other Text Tracks.

The name "audio descriptions" could be confusing since it seems to imply an audio format. This would be the case in Owen's first case, but not in the second case.

Also, I am having a bit of trouble trying to find where the audio descriptions are added in the advanced examples.

The audio description track for that example is found here: https://d2zihajmogu5jn.cloudfront.net/elephantsdream/descriptions.en.vtt

You could check out the Text Tracks guide for more info on that.

[KianBadie]

@misteroneill Ah I see, that makes sense why it wouldn't appear in the Audio Descriptions menu.

Yes I think I may have confused myself. The first example that Owen mentioned is definitely what I was aiming for.

So the only way to add an alternate audio track is through HLS or DASH? I was naively assuming cycling through audio tracks would just switch between different mp4 files. Is the alternate audio track built into the .m3u8 file format? Because that is the only place I could imagine that alternate track being added in the source code of the advanced example (which I found here).

[OwenEdwards]

@KianBadie that's right, the HLS and DASH formats support adding more than one audio track for a single video. In theory, the MP4 format also supports that, but as far as I know, video.js doesn't support multiple audio tracks in an MP4 file (likely because browsers don't support that feature).

So you'll need to look at the HLS source file at the line you highlighted: //d2zihajmogu5jn.cloudfront.net/elephantsdream/hls/ed_hd.m3u8.

You'll need to understand HLS format (e.g. see https://developer.apple.com/streaming/ and https://developer.apple.com/documentation/http_live_streaming/http_live_streaming_hls_authoring_specification_for_apple_devices), but that M3U8 file is in plain text and contains pairs of audio tracks, one labeled "Main" and one labeled "Audio Described" (the second one also has an HLS "CHARACTERISTIC" of "public.accessibility.describes-video", which identifies it as an audio description track in a way that decoders recognize (that's what causes video.js to put the "AD" icon next to that track name in the "Audio Track" menu).

You could add a feature to video.js to support two completely separate .mp4 source files, each with the same video but one with the normal "main" audio and one with the "audio described" audio, and then provide a button in the control bar to completely switch between the two .mp4 source files. But that isn't supported in the core of video.js, and has more issues with buffering than using the HLS format.

[gkatsev]

Safari is the only browser that supports multiple audio tracks in an mp4 right now. Video.js should pick those up. Chrome is working on support for that, but it's not ready yet.

[KianBadie]

@OwenEdwards @gkatsev Ah I see. Thank you both for the explanation! I see now that an understanding of HLS was critical to understand how this feature worked.

I realized that I have been saying mp4 files, but I embarrassingly realized that I meant to say mp3 files. As in having the video still be a mp4 file, but attempting to add a separate mp3 audio track. Is this doable with videojs? Or does switching audio tracks happen only within DASH/HLS/Safari mp4 formats?

To give context, the videos in question are silent video-only content. I'm assuming that doesn't change much functionally, but I hope it shows why I am so hopeful for something easy like adding an mp3 to work!

[gkatsev]

Unfortunately, it isn't possible to add alternate audio tracks purely client-side, at least not without a lot of other work (and it may not work in all browsers/platforms).

Going with HLS or DASH is definitely the recommended approach. With both, you can configure it so that there's a single video track and a bunch of audio tracks. You won't need to have the video combined with each audio.

[KianBadie]

@gkatsev Ah I see. Well thank you all for the help and answering my questions! It was very educational.