Move videos based on music

[nateraw]

Would be nice to add an additional module that would "push" or "jitter" frames of the video based on music. A closed source version of this can be seen here. Would love to have open souce option for doing this with stable diffusion 😄

I don't really have the bandwidth to do this myself right now, so help (even if its just a colab with minimal example) would be much appreciated on this!!

If anybody is creeping on this repo/working on their own project + ends up getting to this before me, please ping me here so I can play with it 😬

[mouhandalkadri]

it's really a nice feature. as a rough idea for the implementation I am thinking about detecting the frequency of the audio clip then changing the interpolation percentage relatively to the frequency changing speed, thus the number of the frames would increase for the low frequency audio. I am planning to use some implementation of Fast Fourier Transform to detect the changing in the frequency. any idea or suggestion before I try working on this ?

[nateraw]

@mouhandalkadri thanks for your interest here! I honestly don't have a great idea on how to do this (and again, have limited bandwidth to think about it), but I did have some ideas. They might be bad haha, but I'll share what I was thinking.

Feel free to ignore anything here and experiment with something else, this is just one idea:

---

Basically, was thinking we keep the frames moving at a consistent rate over time, like we see below...

Then, we set a threshold (let's say -6db). If the audio goes above that threshold, we push frames faster and then come back to join the original rate. So, essentially, you're still moving at the constant rate from above, but you push to-and-from later frames as audio goes above a certain threshold.

It would maybe be useful to apply a low pass filter or something before this (probably down to <1000hz), so it has an exaggerated effect on things like bass/kick drums of house music.

[nateraw]

Relevent: https://github.com/mikaelalafriz/lucid-sonic-dreams

[nateraw]

Update - I've figured this one out. Will post the code when it's clean. Results are 🔥 and will share soon

[nateraw]

Example video here for those following:

https://twitter.com/_nateraw/status/1578049976206237705

Code is still pretty messy and I'm working on fairly large refactor of this repo that will include the changes (along with a ton of breaking API changes). Bear with me while I clean things up - I want it to be really great!

[nateraw]

Closed! Info in readme.