Adding generative effect type

[pj4533]

This implements #59

The idea is to use image2image, but rather than based on the underlying video frame (calling this a 'direct' effect), a 'generative' effect is based on the previous index processed frame. In my CLI I used a slower frame rate and ffmpeg to interpolate to get a nice look, so not sure exactly how I'll finish this, but the groundwork is there.

Todo:

• fix preview
• for generative effects if I dont change the seed it just goes to noise. not sure if there is a better way around that, but for now I just randomly assign a seed that is different to each frame. Currently I don't write these anywhere tho, so I wouldn't be able to recreate a given effect. I should write then to the effect for persisting to disk.

[pj4533]

I am currently blocked on getting the output I really want.

In my old CLI code I used ffmpeg and it's motion interpolation. I would render fewer frames with stable diffusion, and tell ffmpeg that the source PNGs were a lower frame rate, and tell the motion interpolation that I wanted a higher frame rate. This would tell ffmpeg to interpolate the intermediate frames. Giving a smooth output where the stable diffusion frames blend nicely from one into another.

I am not sure how to do this best in swift code though. I was hoping for some Core Image api that would do it, but have found nothing as of yet. I guess my best case for keeping things simple would be to wrap ffmpeg somehow?

I could tell generative effects to render fewer frames, since all that really matters is the starting frame used for the generative effect. Then I just need to fill up the rest of the array of frames for that given effect.

Then I take those PNGs, already on disk, and write swift code that generates a mp4 just as I did before. Then extract the frames from that mp4 and put those URLs in the effect? That might work, but is messy.

NOTE: can ffmpeg output PNGs directly, rather than the mp4?

[pj4533]

The real root of the "problem" is that stable diffusion using image 2 image needs a high enough strength value (enough steps of processing) or the result image is too noisy.

However,in my experimentation, it also needs different seed values or it also just ends up with noise. (Not sure why this is exactly, perhaps I need to learn more about this bit)

So the combination of a fairly high strength and different seed, leads to images that are not similar enough to be in a coherent animation at high frame rates.

The above method gets around this by generating fewer frames (ie a lower frame rate) and the interpolating the intermediate frames, leading to a smoother video output, but still with the generative AI madness that I desire. Lol.

[pj4533]

Other thoughts:

• Could probably remove strength from the new effect UI for generative effects and just use a default value. Similar to seed, changing that has little benefit in this use case.

[pj4533]

This might be the way.

• supports macOS
• compiles to lib(s)
• has a ObjC API (bridge to swift?)
• wouldn't have to go out to command line, nor require brew install ffmpeg
• not sure if is supports motion interpolation, there's like a million different versions
• I am MIT licensed, this is GPL...is that a problem?

https://github.com/arthenica/ffmpeg-kit

[pj4533]

Continuing experiments not 100% sure on the interpolation route. Figuring it out. Some other findings:

• need to add rotate angle and direction setting
• need to add zoom amount setting
• keep strength in for generative effects
• all 3 above probably should be in an 'advanced' settings section for generative effects?
• should I enable rotate/zoom for direct effects?

[pj4533]

After more experimentation, I think the ffmpeg interpolation route is useful, but an optimization. With the right rotate/zoom values, and a good prompt, I can get interesting output at full frame rate. It's flickery, but thats not always a bad thing. Going to finish up this PR, then create a new ticket to implement FFMpegKit.

[pj4533]

First test render went good. Immediate to dos:

• add step count to effect params
• arrange params in new effect dialog better (maybe with advanced section, or maybe even tabs?)

[pj4533]

Check the -1 generative piece with the effect before it...does it do that right?

[pj4533]

Did a test render using generative effect, it works well enough to merge this PR. will need to add some more issues to cover items not included here.