SparseCtrl-RGB, flashing in generated frames

[aihopper]

Here is a minimal repro case.

We'll start with something that works so we can keep it as a reference. This basic workflow below generates a video of a running person (SparseCtrl-RGB module is bypassed so all works as expected):

ComfyUI Workflow	Result

Now I enable the SparseCtrl-RGB module and ask it to interpolate every other frame, notice the flashing:

ComfyUI Workflow	Result

notes:

• the sampling method when generating the reference animation and when interpolating is the same
• the only difference is adding SparseCtrl-RGB
• The flashing happens because the frames generated by AnimateDiff are lighter and have a higher contrast.
• In other repro cases colors shift to orange tones and overall the images degrade.
• A similar thing happens with video prediction.
• Note that the reference sequence was generated by AniumateDiff, this means it should be represented optimally in the latent space.

Any help is ideas on how to debug this are very welcome. Cheers

[aihopper]

Please note this issue also happens in the developers repo: https://github.com/guoyww/AnimateDiff/issues/387

Hopefully this helps others set expectations and saves them some time.