jbilcke-hf / clapper

Clapper.app, a video synthesizer and sequencer designed for the age of AI cinema
https://clapper.app
GNU General Public License v3.0
2.08k stars 194 forks source link

Use a LLM to infer the segments of a scene #78

Open jbilcke-hf opened 2 months ago

jbilcke-hf commented 2 months ago

Context

Currently we use the Broadway library to convert a screenplay to a .clap file

But this uses heuristics (if/elses, pattern matching using regexes) which can scale (importing only takes a few seconds) but are "dumb" and can't get the bigger picture

Solution

Call the LLM when we import a story, for more accurate conversion to the timeline.

Note that this is a non-trivial ticket, there are technical constraints (context window, speed, memory, JSON validation etc)