Integrate a MP4 video assembler directly within Clapper

jbilcke-hf commented 4 months ago

Context

Today there exists a project to convert a .clap file to a MP4 file [code, demo]

It is used by my AI Stories Factory Hugging Face Space

but not by Clapper

Goal

I propose to embed the MP4 exporter directly within Clapper.

Using a WASM port for FFmpeg could be an option I guess (eg. running within web workers, with shared memory access, ability to manipulate large amounts of memory)

If not possible, then on the server-side (eg. generate a .clap in the client-side and send it to the server-side, but this adds some extra delays)

Limitations / things to checks

generating a large project can take a lot of time, so we need a system to keep track of the progression
if we generate on the client side, then we can scale quite easily for peanuts (the same instance of Clapper could server many users), although it may run more slowly (CPU only)
if we generate on the server side, we might need a queue mechanism, and I'm not sure it scales well with large projects/number of users (nobody wants to wait 1h in a queue). On the positive side, we can compile FFmpeg to get GPU acceleration.
FFmpeg is a bit low level, it can do a lot but maybe there are some things that cannot be achieved easily with it (I'd say it's not a deal breaker since there is a separate feature for advanced video editing)

How to implement

To determine, but until then here are some useful links:

FFmpeg WASM: https://github.com/ffmpegwasm

example: https://ffmpegwasm.netlify.app/docs/getting-started/usage#transcode-video-with-progress-experimental

(not sure the WASM version will give us a nice progress update, they warn us about it)

how progressive jobs works in Clapper: look into useIO() there are a few examples of loading something and calling useTasks

    const task = useTasks.getState().add({
      category: TaskCategory.EXPORT,
      visibility: TaskVisibility.BLOCKER, // <- or duplicate the .clap and turn the task into non-blocker
      initialMessage: `Generating MP4 file..`,
      successMessage: `Successfully generated the MP4 file!`,
      value: 0,
    })

    task.setProgress({
      message: "Preparing export..",
      value: 10
    })

Acceptance criteria

We should be able to render a full project to a MP4, with a progress bar (even if it's just an estimate)

lalalune commented 4 months ago

/boss $250

boss-bounty[bot] commented 4 months ago

Total Bounty: $250 Solved by @devniel via pull request: https://github.com/jbilcke-hf/clapper/pull/24 To claim this bounty @devniel must enable receiving bounties at https://www.boss.dev/

Bounties

Bounties paid automatically on close:

@lalalune: $250
Errors
To claim this bounty @devniel must enable receiving bounties at https://www.boss.dev/

jbilcke-hf commented 4 months ago

thank you @lalalune for the bounty! 🤗

devniel commented 3 months ago

Hi,

I'm working on this by using FFmpeg WASM as suggested, seems working well for my test, a short I'm trying to remaster (https://youtu.be/RQYhDdp8F2Q) ; the export will contain empty black color for segments without video content; and the audio will be merged (and trimmed) as the timeline defines it.

Indeed, FFmpeg WASM progress log doesn't work well with concat; but I think it's possible to get it from the raw logs in verbose mode; I could probably finish it tomorrow; WDYT ?

You can check an screencast here: https://youtu.be/XaW5ns2GcJU And the output with audio here: https://www.youtube.com/watch?v=Ya_yz3OGIrY

Thanks!

jbilcke-hf commented 3 months ago

Thank you @devniel!

This is super promising thank you for working on this 🔥 Yes, I think it would be better to have the progress (or logs - the useTasks() hook can also be used just to push status updates even if there is no progress)

jbilcke-hf commented 3 months ago

@devniel let me know if you need help regarding FFmpeg by the way, for instance if you need something changed, if there is a missing API that I should add to Clapper to facilitate your work, or documentation

(this won't impact the bounty)

devniel commented 3 months ago

Hello Julian,

Thanks, for the moment, it's ok on my side; I created a PR with a first version 😎, please check it, I will do some last tests.

Thanks !

jbilcke-hf commented 3 months ago

@devniel

To claim this bounty @devniel must enable receiving bounties at https://www.boss.dev/

jbilcke-hf / clapper