Picture-in-picture interpreting/translation

[NoMoreCRAPTion]

I haven't been able to find out much if PeerTube supports picture-in-picture or not. I know it's possible on Blu-Rays, DVDs, within media players on your own desktop and various news websites which are bootstrapped.

The reason why I am making this request is because PeerTube added language recognition for sign languages in #420 in response to #347.

The issue is that for many signing deaf folks, subtitles and closed captions are not accessible to them. Not everyone is multilingual, or may have asymmetrical fluency. Some kind of sign language implementation is required.

Previous attempts of rectifying the issue

There are attempts to create a unique writing system, but usually everyone goes "well, I can read (blank language), why do I need to learn new writing system?" No one has actually sat down to digitalize the writing systems. There was a volunteer named Daniel Danos working on an open-source font but he vanished without explanation and didn't leave his project files behind for others to pick up.

(For more information on history of sign language writing, including its roots in France see here: https://www.handspeak.com/learn/index.php?id=403)

At the moment, the most accessible option for sign language viewers is to add a picture-in-picture stream of a person in the corner of the screen. You see this throughout the world via government news and such.

Current solutions and their pros/cons

Currently, the best solution is to export the two video files as one as seen here: https://www.youtube.com/watch?v=TCq3ru9HQSc. But this is a short-gap solution and directing certain viewers to a video hosted elsewhere is not inclusive when everyone could be watching the same video with their own needs met within their own clients. You should be able to turn on and off languages you want or don't want, whether it's audible, visual or written.

You can see picture-in-picture in action in this video demonstrating American Sign Language support for Ice Age the movie on Blu-Ray: https://www.youtube.com/watch?v=2lC0uiX_Pz8.

Documentations

Here is the W3C documentation for pre-recorded synchronized media: https://www.w3.org/WAI/WCAG21/Understanding/sign-language-prerecorded.html. The W3C referenced a European initiative from 1998 which can be found here: https://www.sign-lang.uni-hamburg.de/signingbooks/. The project documentation from Germany is really outdated compared to the Blu-Ray inclusion from 2012. And I am pretty sure the world has moved on from using SMIL and RealMedia as well.

There are probably other use cases where this feature would be beneficial beyond the scope of what I am proposing here.

[NoMoreCRAPTion]

Separating this into its own post instead of the main topic since this is just a different person's attempt of resolving the issue:

Source: Able Player

Sign language

Sign language translation is supported in a separate video player, synchronized with the main player. Tips for filming a sign language interpreter are available from Signing Books for the Deaf:

• Filming the Signer
• Editing the Signer

If multiple video sources are already provided (e.g., an MP4 and WebM file), then the sign language video must be available in both of these formats. For each video source that has a sign language version available, add a data-sign-src attribute to the <source> element for that video. The value of this attribute is a path pointing to the sign language version of the video. If a sign language version is available, a sign language button will be added to the media controller. This button will toggle the display of a pop-up window in which the sign language video will appear. Users can move or resize the pop-up window with either mouse or keyboard.

Unfortunately this feature is not currently supported on iOS.

You can see how Able Player implemented the feature here: https://ableplayer.github.io/ableplayer/demos/video10.html

Not quite what I have in mind considering what OBS (Open Broadcaster Studio) and other software are capable of. But it's something.

One should be able to store the video files separately then impose one over the other.

ADDENDUM:

Apparently this is a feature for video.js.

Video.js supports standard kind values for VideoTracks:

"alternative": A possible alternative to the main track. "captions": The main video track with burned in captions "main": The main video track. "sign": The main video track with added sign language overlay. "subtitles": The main video track with burned in subtitles. "commentary": The main video track with burned in commentary. "" (default): No explicit kind, or the kind given by the track's metadata is not recognized by the user agent.

[NoMoreCRAPTion]

Was checking out NRK's website and noticed they have this feature as well.

Getting subtitles and picture-in-picture interpreting is a bit wonky because it requires you to toggle the same-language subtitles first then toggle the sign language option.