search

jbilcke-hf / clapper

Clapper.app, a video synthesizer and sequencer designed for the age of AI cinema
https://clapper.app
GNU General Public License v3.0
2.08k stars 194 forks source link

Closed-caption support #93

Open sendn0des opened 2 months ago

sendn0des commented 2 months ago

Simple closed captioning is a must for social.

This can be done lots of ways and quickly, but "animated" ones akin to Captions.ai would be a big deal

jbilcke-hf commented 2 months ago

There are multiple solutions for this (python or c++ libraries running on server side etc)

But the approach I would propose for Clapper is to generate transparent images and use them as layers (segments of type ClapSegmentCategory.INTERFACE).

They can then be used as overlays when rendering the video preview but also generating actual videos using FFmpeg, or when exporting the project to 3rdparty software (if they support overlay images).

To draw those images we can either use a simple canvas (or html using SVG <foreignObject> eg with this lib)

What is great about this pipeline is that it only bakes (closes) the captions at the very last minute (leaving room to dynamically customize the font family, size, color etc until the user is happy with the result) and it can be extended in the future by generating frames (eg. from a invisible WebGL rendering canvas, to create cool shader effects) to be used as an animated/video overlay.