Feature Request: Scheduling for AnimateDiff

[pksebben]

Variation seeds would be a huge benefit to animation workflows - allowing the precise travel between two seeds in combination with other controls could make it possible to generate slices of video that could later be seamlessly stitched together.

What I'm suggesting is something like the Latent Keyframe in Advanced Controlnet - an override component for Hijack to set the strength based on batch index. Using this, one could set a schedule for the strength over the course of an animation thusly supporting smoother, more colorfast stitching.

[chrisgoringe]

Not familiar with the control you mention - maybe you could sketch out a little more how this could work? Either here, or find me on discord chris.goringe so we can discuss

[pksebben]

Absolutely.

Advanced Controlnet has this concept of 'latent keyframes' - which can be used to set a schedule for the strength of the controlnet.

The way it works is that you set up a series of nodes that determine what strength to set the controlnet to, and at which batch index (there are a couple of different nodes that accomplish this in subtly different ways).

Here's the file with those nodes

Then, in the function that applies the controlnet to the conditioning, you have an override that checks for the latent keyframe group and applies it if present (around line 175):

                            c_net.set_timestep_keyframes(timestep_kf)
                        if latent_kf_override is not None:
                            c_net.latent_keyframe_override = latent_kf_override
                        if weights_override is not None:

I'd have to do a little more digging to figure out where precisely this affects the resulting change (that is, the application of controlnet to conditioning), but I can also imagine a fairly straightforward way to do it with hijack. You check for the batch index and if there's a latent keyframe for that index you apply whatever number is stored in the latent keyframe.

To draft a possible implementation:

@dataclass
class LatentKeyframe:
    batch_index: int
    variation_weight: float

class Hijack:
    #...what's there already

    def func( batch_index, **args):
        if weight_override := get(x for x in self.latent_keyframes if x.batch_index == batch_index, None):
            variation_weight = weight_override
        # ...rest of func()

...or something like that, plus the nodes for creating the latent keyframes and a new input to the hijack node to load them.

[chrisgoringe]

I've retired this node pack. Use https://github.com/chrisgoringe/cg-noisetools