Multi-monitor / multi-track videos

When recording or streaming technical content I often wish there was a way to have multiple monitors shown simultaneously to viewers who also have more than one monitor.

For example, I might be writing some code across two panes on my central monitor, and have reference documentation on my right monitor. Using three vertical columns would be too cramped when combined with the editor font size increase needed to help 1080p viewers read the code. While a viewer could follow along with the code alone, being able to see that reference material on their own second monitor (if they have one) would be ideal.

I can think of a few ways to implement this:

PIP video

The streamed or uploaded video has twice the horizontal or vertical resolution, displaying all monitors in a "picture in picture" fashion. This can be done in OBS with no extra plugins - just change the canvas size and have both monitor capture sources in frame. PeerTube would then have a setting to say "this is a PIP video with this number of rows/cols". A viewer could then click to pop out each, and the frontend can apply a CSS transform to the video element to display the right section.

This is probably the easiest method since it works out of the box and requires minimal changes. It would also be the least likely to have desync issues across the separate players, since it's the exact same video and there's no variation in the position of I-frames.

It does require a specific setup for PeerTube, though - the recorded/streamed video wouldn't work on another platform like Twitch or YouTube, although the user could crop the video for upload manually after recording. It also requires that the whole stack supports sufficient resolution - not sure how well it'd handle video at 7680x4230 for a quad-UHD setup. It's also the most bandwidth intensive for clients since they're streaming all the monitors whether they're looking at them or not.

The video could be split into separate playback tracks server-side to save bandwidth to help alleviate some of the downsides.

Linked videos

In this case the separate monitors would be uploaded as separate videos, or at least separate video files, which would then be linked together as a multi-monitor video. This could also work for streams, if the streamer used multiple OBS instances. The upside of this approach is that multi-platform streaming works the same, and there's no extra effort for uploading elsewhere later. It also has the most seamless experience for users who just watch the "main" monitor - no extra bandwidth, no different to a regular video.

This would almost certainly require more work overall, since PeerTube would need to support multiple incoming streams from the same user, a way to link those streams as playback tracks of the same content, UI work around specifying which video or stream is the primary or default view, and some new backend concepts around videos. It would be the most interoperable approach, though.

Multi-stream videos

In this scenario the separate monitors would be separate video tracks in a single container. I believe RTSP supports this. However, OBS does not support this for streaming or recording, and I'm not aware of any other streaming software that can do this. This would be a nice thing to have standardised across streaming services, but that's aspirational rather than practical for the moment.

I imagine the front-end experience as having two main "modes" when watching a multi-track video. The first would be single-view playback, with a track selection control underneath the video. Clicking a track swaps to that video. There could also be keyboard shortcuts for next/prev. This mode would be used by people who just have one monitor, or only want to use one monitor. For the PIP option this would just apply a CSS transform and be an instant swap. For other options it would incur a small delay for re-buffering, although a future release might offer an opt-in option to buffer all streams so that change-over is immediate.

The other mode would be multi-view playback. There would be pop-out links for the alternate monitors, which would open a new window/tab to the the same video page but with a URL parameter specifying which track to play. That way the separate tracks can be opened on separate monitors. Synchronisation and play/pause/scrub events across windows/tabs can be managed via BroadcastChannel.

A future upgrade to this could be a way to mark the active video track, e.g. via SET_PARAMETER in RTSP, so that streaming software (e.g. an OBS plugin) could send that information during a livestream (e.g. via mouse tracking) so that users have an indication of which monitor is the "current" one. This could also work with chapter markers in video files, or a plaintext timestamp file.

I took a look around the PeerTube source tree to see if this is something I'd have any luck in implementing myself, but unfortunately I'm a bit out of my depth with a project this size and a feature that touches a lot of different areas. Still, I'm keen to help in any way I can.

Chocobozzz / PeerTube