[BUG] animatediff?

[dchatel]

After extensive tries with animatediff, I'm a little bit confused.

Sometimes it works, sometimes it doesn't. AREA never works with animatediff, it fails with an error. MASK sometimes work, sometimes doesn't. Sometimes, you get only noise, sometimes you get a succession of images with zero temporal coherence, sometimes you get an animation, but its content (relative to the mask) seem to be ignored, and sometimes it just works.

The most amazingly incomprehensible is that this weird behavior seem to be dependent on changing any parameter is the workflow.

For example, using this workflow: weird.json You get this: AnimateDiff_00035.webm which is pretty amazing.

But the prompt was:

couple, marriage proposal 

AND MASK(.65 1, 0 1, .75)
female, smile, (dress:1.3)

AND MASK(0 .35, 0 1, .75)
one_knee down, bearded male, casual clothes

So the male and female characters position are switched.

Then you hit queue prompt again, without changing anything (so the only thing changing is the seed), and you get this: AnimateDiff_00039.webm No temporal coherence.

Now you change the prompt to just add a single space anywhere: AnimateDiff_00040.webm

Queue prompt again, without any other change: AnimateDiff_00041.webm

sqrt_linear beta_schedule to lcm: AnimateDiff_00044.webm

queue again: AnimateDiff_00045.webm

animatediff end_percent from 1 to 0.999: AnimateDiff_00046.webm

etc... Changing the resolution has no effect on this weird behavior.

What's going on here?

[asagi4]

I don't use animatediff myself so if the nodes work it's purely by luck. It might be that adiff is doing something weird with conds or has its own sampler overrides that conflict with the monkeypatch that my node uses.

I suppose I can take a look at some point if I can figure out what is going on.

[dchatel]

Apparently, there's also a weird interaction between your nodes, smzNodes and animatediff. If you have smzNodes installed (even if you don't use them), the first generation on some workflow might trigger an error. Then the second will generate noise. And only the 3rd will succeed.

[asagi4]

Yeah, that's probably a monkeypatch interaction.

Lots of nodes have to do hacky stuff to hook into sampling because ComfyUI's native hooks are pretty limited and those hacks can conflict with each other.