Use case description: responsive grafics synced with video

[pthopesch]

I'd like to describe a use case we have right now that could make use of the DataCue proposal.

We are aiming to install a workflow for responsive grafics in VoD content. The aim is to transport grafical elements/animations separately to the client device and dynamically compose the final image in the player application. I'm talking about typical TV grafics here, including even a broadcasters logo. One of the main scenarios where this seems promising for us is adapting grafics for a portrait mode playback on mobile devices. In some cases, the syncronisation between the video image and the grafics must be very accurate. I guess the use case is similar to subtitles in that regards.

There is a plattform called Singular.Live that already does something like that on a proprietary basis. I'm sure there are also other companies/solutions.

But I don't know of any (implemented) specification that allows synchronising the grafic playback with an HTML5 video element. We could use a TextTrack of course. The DataCue seems to provide exactly that. Has such a use case been discussed already in this group?

Of course, if somebody is working on a similar topic, it would be great to share experiences here.

[ingararntzen]

Hi.

I'd recommend the https://webtiming.github.io/timingsrc/ framework.

It's an open source JS library that allows you to synchronize anything with anything on the Web - including HTML5 video and timed data (i.e. cues) with precision in the single digit milliseconds.

You would likely want to use the sequencer to control the timing of the graphical elements. It works similar to the texttrack mechanism - except it's much better (disclaimer - I made it).

In terms of standardization, the TimingObject model (which the framework is based on) was proposed for Web standardization a few years back by the Multi-device Timing Community Group.

If you find it valuable, let us know :)

[pthopesch]

Thank's a lot for this info. It seems the main purpose of this framework is to provide synchronization between different devices, based on a worldclock, correct? That would of course work locally as well, I'll take a closer look at it.

[pthopesch]

As discussed in yesterday's Media Timed Events / WICG DataCue API call, here is a short use case description that may be included as an additional use case in the IG Note:

Media stream with video and synchronized graphics

A content provider wants to provide synchronized graphical elements along with a video. Graphics and video are delivered separately to a client device, either in-band (i.e. multiplexed with the media stream) or as a sidecar file. The graphical element stream or file contains media timed events for start and end time of each graphical element, similar to a subtitle stream or file. A graphic renderer takes the graphical element stream or file as input and renders it on top of the video image according to the media timed events.

The purpose of rendering the graphical element at the client device may be that the graphic renderer may optimize it for the devices display parameters like the display aspect ratio and its orientation. Another use case might include localization of the graphical element.

The synchronization between video and graphics should ideally be frame accurate.

[ingararntzen]

Thank's a lot for this info. It seems the main purpose of this framework is to provide synchronization between different devices, based on a worldclock, correct? That would of course work locally as well, I'll take a closer look at it.

@pthopesch Not exactly right - but close :).

The main purpose is to provide a local media clock/controls for the Web (a.k.a timingobject) that is an independent mediator between timed things, and designed for precise synchronization.

The html5 media clock/controls which is typically used for this fails on both these accounts.

Once you have a timingobject as shared concept for media clock/controls, you can have different timed things be precisely synchronized simply by connecting them to the same timingobject. Real easy.

The inter-device sync is merely a secondary benefit (yet substantial!), as the timing object can additionally mediate between timed things on other web pages and/or other devices.

Note: this approach works well if you have time data delivered out-of-band. If data is delivered in-band, I'm not sure if you can escape the datacue/texttrack system, and you are effectively stuck with the video as your media clock. This is yet another reason to avoid in-band transfer of timed data, in my opinion.

Ingar