kijai / ComfyUI-DynamiCrafterWrapper

Wrapper to use DynamiCrafter models in ComfyUI
Other
489 stars 15 forks source link

tooncrafter is fading in/out between images #66

Open 3blackbar opened 3 weeks ago

3blackbar commented 3 weeks ago

I gave 4 input images, on huggingface i do this then theres no blurriness cause it goes from image 1 to 2, so how to make it identical her ? I dont want fade into image2 then fade into image 3 etc... each image fades into another and its definitely not what99% of the users would want. How to disable this? so it gets me exactly the output thats on reference image 2 ./3 /4 and not fadeed out 1image1 and 2 and 3 and 4 where images are spit.

Any way to let it render 1 to 2 only, then 2 to 3 only, then 3 to 4 only ? So there is no fade in/out cause it does not have image to fade into at the end ? And i still would like to feed it multiple images not just 2 cause i know that if ill do it with 2 then i will get no fade in/out. A switch would be nice.

Yeah image 1 to 2 is great but image 1 to 2 to 3 is pretty bad cause it fades into 3rd image, can someone modify the node so it does not fade ?

Here is example, you can see how 2nd framew and 3d frame are TOGETHER in generated output - this is very wrong. 0afe29b452

And When it goes from 1st to 2nd to 3rd it should not keep 2nd image in generated frames.

These are input frames e3 e2 e1

The best result i get when i do 1 to 2 , save it , then do 2 to 3 , save it , and 3 to 4 , save it , and manualy merge them append one after another and remove last or first frames of each segment as theyre indentical on end and beginning , i wish we could apply that workflow to this repo as well, it gets me this one below

exp1

Also when i use less than 16 in tooncrafter node for frames amount under eta - it gives me cracked noise .

kijai commented 3 weeks ago

When you input more than 3 images, the node is doing 1 to 2, then 2 to 3, and then joining the results. There is no interaction between 1 and 3 whatsoever. The results vary per seed so much that you will sometimes get fades or otherwise bad results. This is why it's best to do them separately like you said, so that you can redo the transitions that don't work without having to do the whole thing again.

Any other frame count than 16 doesn't work with the current model, it's trained to work with only that.

I don't know what node you mean by the resizing issue?

kijai commented 3 weeks ago

https://github.com/kijai/ComfyUI-DynamiCrafterWrapper/assets/40791699/0fd432cd-6f18-4b93-8a55-9c1f709fd19d

3blackbar commented 3 weeks ago

yeah i did it like you before i read that and it is fine now theres no fade ins. But huggingface worked with 10 frames ? IS thre a chance to allow the node to accept lineart and color image like their demos did so it will do full color scene ?

kijai commented 3 weeks ago

The model itself can't work for 10 frames, it always has to generate 16, but it's of course possible to drop frames after the generation to retime the animation to your liking, I'm hoping to make some helper node for that.

They have not released the code for the sketch guidance or the colorization yet, still waiting on that to implement it.

3blackbar commented 3 weeks ago

yeh i noticed that it just drops the frames on huggingface, and its the reason i got crap outputs sometimes. Thanks for the info on sketch guidance, i hope they do and higher res model, now my workflow is tooncrafter then SUPIR to upscale cause it sticks to original lowres shape very well unlike other upscalers with img2img.

For generating less frames i think RIFE or FILM could help to generate inbetweens and then its easier to drop particular frames and still get smooth motion cadence But generating 16 and giving option to save just 8 should be straightforward, or even 4. or maybe even 5 would be acceptable since its just one frame from 15