Synchronize video/audio playback with multiple concurrent users

[Tzahi12345]

This thread on reddit got me thinking, how easy would it be to implement something like SyncTube, where you can sync playback and have users watch a video at the same time?

The way I see it, there are two requirements to this:

1. Ability to match the basic playback state from the owner's stream (consist of playing or paused, and a timestamp)
2. Resync requests from the client at a specified interval that will fix desync issues

The backend would need to keep track of the state of each of these streams in a stream state object.

ex.

{
  "unix_timestamp": 1594966671.000 // unix timestamp i.e. seconds since January 1, 1970
  "playback_timestamp":  7.912 // where the user is in the video, in this case 7.912 seconds in 
  "playing": true // whether the video is playing
}

Why is this information necessary? Because the user ultimately needs to know whether their stream is synchronized, and using those variables above, we can determine the error. Let's say they send a resync request, and in the request they send their unix timestamp of 1594966672.500 and video playback of 9.512.

The difference of the unix timestamp is 1.5 seconds, while the difference in the playback timestamp is a whopping 1.6. That means that the watcher is 100ms ahead of the streamer. So the watcher should move 0.1 seconds back in its playback position to match the state of the streamer.

In this case, 100ms is relatively minor and we can probably avoid the resync process. But sometimes this delay can be significant enough that it should be adjusted, we'd probably need to do testing to figure out that threshold.

The streamer will have to continuously send updates so that the state in the backend matches that of the streamer. If the stream gets paused, the stream state in the backend will reflect that. The above math will be skipped (as sync is unnecessary here), instead the watchers playback position will simply match that of the stream state and wait for the stream to begin.

Streams stop when the streamer hasn't sent an update for a certain period of time. It will be assumed that the streamer closed the tab. The streamer's player component will have a button to start and end the stream by themselves at any point in time.

Sounds feasible? Am I missing something obvious? Curious as to what you guys think. Let me know, thanks!

[GlassedSilver]

Just recently I thought about how amazing this would be, to open my server (on-demand for my taste, but whatever rocks your boat!) for WAN access to have shared sessions with my SO.

ESPECIALLY useful when you want to share the experience for videos not available anymore online or stuff like manual submissions. #157 comes to mind

[GlassedSilver]

Instead of skipping to or back a position I guess adjusting the playback speed for a short time (possibly for both clients) would be better.

This avoids stuttery playback.

SyncPlay in Jellyfin works like that afaik.

[Tzahi12345]

This avoids stuttery playback.

You're right, but it only happened once I started recording the following demo (oh, and btw I got it working kind of!):

https://user-images.githubusercontent.com/707413/111569899-4393af00-8769-11eb-9cb6-7b230e2c3a86.mp4

So on low-performance systems it's an issue, on my mid-range laptop it handled it fine though, until I pulled out OBS. It's definitely possible to just speed up the playback but it will require some more math and if statements. Nothing crazy, I'll see if it's a simple task as I move along on this feature.

Note that the player on the left is the "streamer", the one on the right is not even logged in and is simply watching.

There is still some more logic to work out, like only showing the "join stream" button (not visible in the video) when a stream is available, and what to do when no updates are received for a certain period of time.

[GlassedSilver]

It would be awesome if the group could stay together even after a play finishes, so you don't have to rejoin every time a new video is selected.

Maybe also staying on the same playlist could be supported?

If this is going to become a feature, maybe we also need permissions. Who can add, who can skip, etc....

Something else I noticed... Your view count is at 21 for that video... I want to re-iterate how important it would be to count views and plays differently.

Right now I'm avoiding opening many videos, because I want their "view count at 0" being my unwatched marker - so that I watch them later. This would especially be important if I let the instance (through my personal VPN connection) be shared with friends who will increase the view counts.

Just saying. :P