Globally synchronized playback

[itsjamie]

When playing media from a stream source like RTMP, you can more easily get synchronized playback between multiple viewers, however, the infrastructure cost can make these "active" socket streaming technologies cost prohibitive. More and more, we're seeing the CDN networks move away from RTMP, and towards the adaptive HTTP models such as DASH or HLS.

This makes a "room synchronized" playback cost prohibitive, or require a slightly different approach.

There is an interest we're seeing in the ability to synchronize playback of content in the <video> element based off timing sources like the HLS #EXT-X-PROGRAM-DATE-TIME, or DASH's AvailabilityStartTime. I had a brief conversation with @chrisn on this, and we discussed which portions could be part of a specification, and what couldn't due to the current playback state for these formats and the reliance on MSE (which is great!).

I thought a little on it, and I think that almost the entire thing could be standardized if the video element added the concept of a TimeSource. It could be used to provide a fallback for the wall-clock implementation based on information presented from external sources like hls.js or dash.js. This TimeSource could then be used alongside the PTS information available to the <video> element to synchronize playback to a wall-clock.

For example (psuedo code):

video.timeSource = { 
    // converts video currentTime to a wall-clock Date value to return for MSE implementations
    getCurrentDate: function(currentTime) { return dash.wallClock(currentTime) } 
    getTime: function() { return Date.now() } 
    /* 
    returns the wallclock implementation for DateTime, could be a NTP implementation, 
    or extrapolate from the information in the Dash or HLS stream to return the live edge 
    date, or use something like <UTCTiming> from Dash. 
    */
} 
video.playSynchronized(3)

This would attempt to always bring clients to playing content three seconds from the live edge. There are various techniques that could be applied like varying the playback speed in small increments to bring the user into the time range, but not so much they can hear or see it happening.

Thank you!

[itsjamie]

Tagging some great folks from Akamai for visibility and assist. They're on the CDN side of some of the work for the current project that is bringing some of this to front-of-mind @wilaw @amoening.

[ingararntzen]

Hi @itsjamie

I thought you would like to know that we've had an initiative going for a few years calling for Web support for globally synchronized playback. You're more than welcome to join in :)

The Multi-device Timing Community Group was chartered with this very purpose. We've also published a draft specification for the Timing Object which incidentally suggests a timingsrc property for media elements ;) Additionally Timingsrc is an open source implementation of a programming model around the timing object, including functions for synchronization of media elements, timed data, and integration with online timing providers for global playback. The site also has documentation, example code and live demoes.

Here's a demo we made before using timing objects for globally synchronized playback of DASH a few years back (without making any modifications to the dash player!). Live synchronized DASH

Note also that the scope of the timing object proposal is a little broader than what you suggest here. For one thing, its not only for sharing a reference clock, but also targets active control (e.g. pause, skip, fast-forward) globally. Furthermore, its not limited to media elements, but targets synchronization of anything timed. Still, the use case with synchronizing CDN playback is certainly an important use-case and one that we hope could attract some support from CDN's. I think your use case fits nicely with the timing object proposal. What you suggest as timeSource is somewhat similar to a timing provider in the timing object model.

Again, I hope you'll take a look at the multi-device timing community group, and hopefully join and help out :)

Best regards,

Ingar Arntzen, chair Multi-device Timing Community Group

[chrisn]

Related to this is a new WebKit proposal for a Media Session Coordinator API to enable synchronised playback across devices. This appears to be built on the Group Activities framework on Apple devices - see this video from WWDC 2021 for more on that. (Interestingly, the WebKit explainer references the Timing Object proposal).

[ingararntzen]

Great to see that Apple is pushing for multi-device sync !

Also great that they are referencing the Timing Object proposal :)

Interestingly, there seems to be a little confusion as to how these proposals relate to each other. Specifically, the Webkit explainer lists the synchronization mechanism as a non-goal, and points to the Multi-device Timing group for a solution for this mechanism.

However, the Multi-device Timing group also defines the sync mechanism as a non-goal -- as it is not clear that standardization will be needed for this function. Instead, the timing provider interface was proposed to allow timing objects to use different services for synchronization.

So, the two proposals certainly share some goals. I see the Media Session Coordinator API as shaped for a specific application type. It is also not geared towards precise sync (a non-goal).

I expect it would be easy to implement the Coordinator API on top of timing objects. This would make it easy to sync with non-media things as well.

In any case, with Apple pushing for this, I'm hopeful there could be some renewed interest for this within the Media and Entertainment IG :)

[chrisn]

It would be interesting to know how much interest there is from other content providers. One of the things we did at the BBC in response to the pandemic was to create https://www.bbc.co.uk/together, to enable shared synchronised viewing experiences.

[chiefcll]

Comcast also did this and called it Watch Party. It wasn't widely marketed and currently unavailable.

[ingararntzen]

Yes. Facebook has a watchparty thing too. Frankly, I find the user experience of these things to be not so great - forcing viewers to leave familiar interfaces/services and start minding things like login and sessions and group memberships and chat and all these things. As such, I would not be very surprised if viewer feedback was lukewarm and content provider interests correspondingly reserved...

How about something convenient? Say just making it a feature of the regular offering - just make live Web TV synchronized always (like TV used to be) or optional by toggling on or off? What about just sharing a link if you want to watch VoD content with someone? This would be invisible for users that don't care for it, and easy enough for those who actively want to engage.

That said, I don't think collaborative viewing even makes my "top 10 uses of global synchronization" - so estimation of content provider interest should ideally have a broader scope than just this one use case.

[chrisn]

forcing viewers to leave familiar interfaces/services and start minding things like login and sessions and group memberships and chat and all these things

From what I can tell, the WebKit solution builds on native features that implement all of that, so is presumably tied into one vendor's solutions?

What about just sharing a link if you want to watch VoD content with someone? This would be invisible for users that don't care for it, and easy enough for those who actively want to engage.

The BBC solution works in this way.

That said, I don't think collaborative viewing even makes my "top 10 uses of global synchronization" - so estimation of content provider interest should ideally have a broader scope than just this one use case.

That's fair. Do you have a list written somewhere?

[ingararntzen]

About the top 10 list - that was figure of speech ;)

Still, the use-case section [1] of the timing object draft lists a number of relevant cases.

The timingsrc project also has an introduction [2] which highlights the importance of both local and global synchronization.

However, co-presentation of timed media content requires fairly precise synchronization, and the Web has little support for this. The Web is primarily a platform for embedding independent media frameworks. It offers no particular mechanism for precise coordination across different media frameworks.

As gaps go, this gap is pretty significant. And to be frank, rather embarrassing too. The Web is arguably the most important multi-media platform world wide. Yet, ironically, multi-media playback is largely delegated to embedded frameworks, and cross-framework playback is simply not supported.

Otherwise, there are many use cases for global synchronization outside the realms of entertainment media. For instance, critical infrastructure for situational awareness, search and rescue etc are increasingly embracing web media technologies for capture, distribution and flexible presentation of all kinds of data, including sensor inputs, camera angles, microphones, text input. What they all need though is consistency in presentation, so that rendered presentation correspond real-life event sequencers. Network and processing delays will quickly mess with the ordering of things, so simply pushing data through real-time distribution systems and hoping for the best, makes the resulting experiences confusing at best, life threatening at worst.

If I need to pick one, this is probably my favorite use of global synchronization, though I suspect the need to capture distributed events and reproduce consistently in Web-based interfaces is really something that is just needed everywhere.

[1] https://webtiming.github.io/timingobject/#use-cases-and-requirements [2] http://localhost/timingdocs/main/intro.html

Cheers.