Subtitle support

[rafael2k]

How do you think we could add support for subtitles? May be support for WebVVT (S_TEXT/WEBVTT) tracks as specified in: https://www.matroska.org/technical/subtitles.html

I know this is long shot, just starting the discussion here...

[Vanilagy]

I know it might feel like a long shot, but I think this is the natural next step for a library aiming to provide a complete muxing experience - so thanks for bringing it up! I'll read into the matter when I have time, but if it's just about adding a WebVVT file (maybe split up into segments) into the muxed file, that sounds very doable. Creating the WebVVT itself would not be in the scope of the muxer, the same way encoding video isn't.

[rafael2k]

Cool! Strange enough, WebM seems not to support subtitles by official spec, but at the same, their website comes with a full sub-chapter explaining how to add a subtitle track... https://www.webmproject.org/docs/container/#webvtt-guidelines

SRT subtitles seems the easier to add on the other hand... as it is just S_TEXT/UTF8 in the data block with correct timestamp set.

[Vanilagy]

Yea, I've read that it now supports WebVTT only. Seems like a good move to allow any text codec and just encourage the user to use type 'matroska' for everything that isn't WebVTT.

The WebM support thing is wack anyway, since I assume most players just use their MKV player for WebM files anyway. But still, important to at least tell the user what's right.

[Vanilagy]

If you feel up for it, you could already try and play around with adding subtitle support to the muxer. I was thinking a method addTextChunk might be good, along with a text track config in the options. Alternatively, we could call it Subtitle instead of Text.

[rafael2k]

My knowledge in JS is very basic... I trust you whatever solution you think it is better! And I agree... most of the demuxers are based on the full matroska demuxer anyway, so there is a high chance that all the common subtitles types might just work.

[Vanilagy]

I see! I'll look into it when I got the time. Which languages do you usually work with?

[rafael2k]

Mostly C. But I'm helping in a JS audio editor, which is pretty cool indeed (and I'm using webm-muxer): https://github.com/ColmenaDev/colmena-waveform-playlist/

[Vanilagy]

Looking into it, adding subtitles does seem like a good amount of effort. Adding the individual Blocks for subtitle tracks is easy, but you also need a way to go from a full subtitle document to these blocks, per subtitle format. I could write an additional VTT helper library to assist with these tasks, but I'm not sure that I'd want to include that in the main library.

All of this is definitely possible, but I'm not sure if it'd be worth the effort as it's quite a niche feature - therefore I'll be closing this issue for now.

If you happen to require this feature urgently and are willing to invest in its development, please feel free to reach out to me!

[rafael2k]

I agree. Lets keep this on hold for now.

[rafael2k]

My idea with subtitles was to add export feature with subtitles for an audio editor: https://naomiaro.github.io/waveform-playlist/annotations.html

[justin0mcateer]

I hate to necrobump here, but we are in need of subtitle support and would be willing to invest some effort into it. You mentioned adding VTT support as a separate library. Do you have any ideas/recommendations on how such an external add-on could integrate with webm-muxer?

[Vanilagy]

No shame in necrobumping! Only problem is that I've forgotten most of the details of my reasoning here, so let's hope I can remember it somewhat.

I think the problem was that WebVTT subtitle tracks, just like video and audio, needs to be chunked before being included in the final file. This chunking happens naturally when using video and audio encoders, but there is no "subtitle encoder", therefore there would have to be additional code which chunks a WebVTT file into smaller segments. I remember that this felt out of scope for this library and therefore closed the issue.

This external library would have to play the role of a subtitle encoder, chunking the file into smaller blocks which would then be sent to the muxer via a method like addSubtitleChunk, addTextChunk or whatever.

Since it is still quite a niche feature, I'd be willing to add subtitle support to webm-muxer for $100. Is that okay for you?

[justin0mcateer]

Yes, we would be interested in that arrangement. When you say 'add support', what would be the scope of that? Adding the chunking logic and everything or adding the 'addTextChunk' method only? We would be getting the text in chunks in near real-time anyway. Would the chunks have to align to certain time boundaries? If so, how would we know what those are?

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Vanilagy @.> Sent: Friday, December 22, 2023 10:05:28 AM To: Vanilagy/webm-muxer @.> Cc: justin0mcateer @.>; Comment @.> Subject: Re: [Vanilagy/webm-muxer] Subtitle support (Issue #11)

No shame in necrobumping! Only problem is that I've forgotten most of the details of my reasoning here, so let's hope I can remember it somewhat.

I think the problem was that WebVTT subtitle tracks, just like video and audio, needs to be chunked before being included in the final file. This chunking happens naturally when using video and audio encoders, but there is no "subtitle encoder", therefore there would have to be additional code which chunks a WebVTT file into smaller segments. I remember that this felt out of scope for this library and therefore closed the issue.

This external library would have to play the role of a subtitle encoder, chunking the file into smaller blocks which would then be sent to the muxer via a method like addSubtitleChunk, addTextChunk or whatever.

Since it is still quite a niche feature, I'd be willing to add subtitle support to webm-muxer for $100. Is that okay for you?

— Reply to this email directly, view it on GitHubhttps://github.com/Vanilagy/webm-muxer/issues/11#issuecomment-1867850353, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAT4Q7OBJN6EA3LNW54CC3TYKWVURAVCNFSM6AAAAAAVQCQZI2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRXHA2TAMZVGM. You are receiving this because you commented.Message ID: @.***>

[Vanilagy]

Within the $100 I'll include the chunking logic, I think I'll just add it to this library unless it grows way too large. What did you mean with getting the chunks in real time, by the way? How are you getting them?

You can donate to my Ko-fi here: https://ko-fi.com/vanilagy Once I get the payment, I'll get to it as soon as I can ^^

[rafael2k]

Hey! Nice to see activity here! We are still very interested in adding subtitles support tru wem-muxer for audio+subtitle export in our audio editor: https://github.com/ColmenaDev/colmena-waveform-playlist

[Vanilagy]

Once it lands, we should make sure it works for your use case @rafael2k

[justin0mcateer]

@Vanilagy We would be recording communication sessions in near real-time. We could have multiple text streams, which would come from the application. Examples might be: a stream for transcription of audio, another for textual user generated messages.

[justin0mcateer]

@Vanilagy check your Ko-fi

[Vanilagy]

@justin0mcateer Thank you!! I'll keep in touch with you.

[Vanilagy]

After a decent grind, subtitle support has landed in v3.2.0! Please check it out and test it.

Took a bit longer than expected but I wanted to get it right. I kept the API as close to the other methods and WebCodecs API as possible. Check out this example from the README:

import { Muxer, SubtitleEncoder, ArrayBufferTarget } from 'webm-muxer';

let muxer = new Muxer({
    target: new ArrayBufferTarget(),
    subtitles: {
        codec: 'S_TEXT/WEBVTT'
    },
    // ....
});

let subtitleEncoder = new SubtitleEncoder({
    output: (chunk, meta) => muxer.addSubtitleChunk(chunk, meta),
    error: e => console.error(e)
});
subtitleEncoder.configure({
    codec: 'webvtt'
});

let simpleWebvttFile =
`WEBVTT

00:00:00.000 --> 00:00:10.000
Example entry 1: Hello <b>world</b>.
`;
subtitleEncoder.encode(simpleWebvttFile);

// ...

muxer.finalize();