WebVTT Subtitle Video "Multiple Annotations", Customized language, and Multiple Events

[njss]

I am deeply interested in using VTT subtitles to annotate videos from experiments. Some of the main functionalities needed are:

• the possibility of defining a "new" custom language besides using the standard set (English, Portuguese, ...). The reason is that I might need to code my annotations and translate them later, on the fly, from a database to a standard/normal descriptive subtitle text language
• The need to encode multiple ongoing actions (for example to describe multiple body movements currently being performed on the video by the actors)
• The need to play this customized language encoded annotations (subtitle) and create on the fly the real subtitle text that should be played in different languages, and the need to capture events from the multiple body parts annotations in real-time so that I can associate these events (and coded annotations) with sensor data corresponding to the annotated events

I did a small test, by creating a VTT player in HTML5 and capture the events, it seems to work, however, I would like to be sure of the best way to address this use case (for which I strongly believe that VTT would give a great contribution, for example, to develop a learning system that also can make use of human behaviour sensing in real-time)

Apologies if here is not the best place to ask these questions. Any comment will be highly appreciated!

Thank you,

Nelson

[fsoder]

Naming "the best way(TM)" to do something tends to be a tall order, but here are some suggestions/remarks. (I'm assuming an HTML environment here but I don't think it should be too difficult to translate to something else.)

It sounds like what you may have/want/need is a "metadata" track. The cue payload can essentially be anything, so if have a "DSL" for defining the events you can encode it and store it as the cue payload, leaving timing to the cue. That the payload is not a natural language doesn't really matter. How you want to encode actions is really up to you - one or multiple events/actions in one cue etc.

For the actual display you could dynamically create a track (addTextTrack) and populate it with (decoded/translated) cues on the fly based on the events (cues beginning/ending) from the metadata track. (Not sure I understood if you wanted real-time capture as well, but I suspect then you'd just get streams of events from different sources.) Issues that could arise here is things like event delivery latency, which depending on QoI could vary.

[njss]

@fsoder Thank you so much! This is exactly what I am looking for... Just trying not to reinvent the wheel...on something that is already implemented! These are good new directions.

Best, Nelson

[gkatsev]

Going to echo @fsoder here. kind metadata is what you're looking for. You can then set the mode to hidden so that you get events but not actually display it. Though, technically, you can do that with any track as well.

[njss]

@gkatsev Thank you so much. I will try it now!

[njss]

@gkatsev Yes...it works like a charm! Thank you.

[gkatsev]

Glad to hear you got it working!