hithereai
commented
1 year ago This is not something we plan to do, nor it sounds like a good idea to approve every single frame manually.
You might want to use @deforuma, but note that until we implement it more deeply into our code, once you use their code, there is no support from deforum, at all.
We plan to integrate it but it takes much more time than we initially thought. So no estimations on time for now.
Is there an existing issue for this?
What would your feature do ?
Stable Diffusion output is still quite unpredictable and even when we do everything right, we still get bad output, ruining an otherwise amazing image. These issues become multiplicative in Deforum, where you're essentially running a batch output of hundreds of images. The 'Redo generation', 'Reroll blank frames' and 'Coherence' features help a LOT, but even doing everything right, an animation can still go wrong and once once it does, the job is ruined and there's no way to to fix it aside from cancelling the job, then manually editing the settings file, setting it to the frame it went wrong, then rerunning the job using 'Resume from timestring'. Doing this every time something goes wrong is impractical. And what if it's only a few bad frames, here and there? Personally, I go into PhotoShop and retouch the bad frames, or bring them into Inpaint, but what if we could just replace them during the job with better ones?
Working towards one-shot error-free videos, let's add a mode where you can 'Approve' or 'Redo' EVERY FRAME yourself. You could even have a mode where you choose the best result from 3-5 images for each frame, just like 'Redo regeneration' clicking on the one you like best. Another use would be Pausing the job just before a major transition/change, switching to 'Approve Mode', then changing the settings and once you know it's going right, switching back to regular mode.
Lets say you found an amazing prompt/setting combo, but 1 out of every 30 frames you generate with your settings give you a mutant double head and all the negative prompts and textual inversions you use can only get it down to that number. Currently, you might fix the bad frames or perhaps give up and try another setup, but with this handy-dandy feature, you could reroll these occasional bad frames, so your animation is one-shot perfection! And what if only 1 in 30 is actually good, but it's AMAZINGLY GOOD? Currently, there's no way to turn that into an animation, but with this, you could just spam 'Redo' until you got what you wanted.
The feature I described above will be easiest to implement, but once it is working, you could have an even more powerful feature I might call 'Review Mode', which allows you to go through a generated video, frame-by-frame and reroll JUST the frames you don't like. And if that's not enough to get you excited, imagine changing one frame, then having Deforum 're-flow' the following frames with latent blending.
I could use this feature in nearly every animation I make! It would go a long way towards making SD + Deforum an indispensable tool that you can get predictable results from.
Proposed workflow
Additional information
No response
Are you going to help adding it?
I'm always here to help. Email or DM me on Discord Vendacious#6006.