About time embedding in ReferenceNet

[RedRAINXXXX]

In the official paper, the authors say

While ReferenceNet introduces a comparable number of parameters to the denoising UNet, in diffusion-based video generation, all video frames undergo denoising multiple times, whereas ReferenceNet only needs to extract features once throughout the entire process

But in your implementation of inference, the forward of ReferenceNet is performed multiple times.

Consider fixing the timestep of ReferenceUnet？

[guoqincode]

At each timestep, reference net is executed only once instead of n_frame times.

[RedRAINXXXX]

What I mean is that referenceunt should do reference once in the denoising process, and these features will be reused in all timesteps of denoising unet. But not “n_frame times”?

[guoqincode]

You can email me at guoqin@stu.pku.edu.cn and we'll talk about this in more detail!

[RedRAINXXXX]

I have sent you an email, please check it~