Video captions

[swissspidy]

Ability to create video subtitles on-demand.

Unfortunately it's not possible to do this automatically for new uploads, as the upload queue has no knowledge about which exact block the video is for, and can't update its attributes. And the onChange callback doesn't update the tracks either: https://github.com/WordPress/gutenberg/blob/4494a79e24bfbbcda97ce9af5db3dcb9e81b09f6/packages/block-library/src/video/edit.js#L133-L156

[swissspidy]

And the onChange callback doesn't update the tracks either: WordPress/gutenberg@4494a79/packages/block-library/src/video/edit.js#L133-L156

Maybe GB could, in a backward compatible way, provide the clientID to the upload function somehow.

[swissspidy]

There's also https://github.com/xenova/whisper-web, though it seems more useful for transcripts

[swissspidy]

The current solution seems to work fine, though needs some fine-tuning to ensure proper length (x words/characters per line). Not sure if it does proper sentences to split by, but that could be an option.

[swissspidy]

Would be good to compare the current solution with https://github.com/xenova/whisper-web. The upcoming release there might have better support for word-level timestamps, which would help for subtitles, but even without that it seems to have good output already.

[swissspidy]

vosk-browser is quite heavy (5.5MB). There is vosklet, which is a bit smaller (3.5MB)

[swissspidy]

Once captions are generated, an editor a la https://vtt-creator.com (https://github.com/roballsopp/vtt-creator, no open source license) would be nice. https://github.com/opusonline/webvtt-editor could be a starting point.

[swissspidy]

Apparently the fastest solution at the moment: https://github.com/FL33TW00D/whisper-turbo