question about forward_cond

SUDO-AI-3D / zero123plus

Code repository for Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model.

Apache License 2.0

1.56k stars 108 forks source link

question about forward_cond #58

Closed ZZWENG closed 5 months ago

ZZWENG commented 5 months ago

Hi, thank you for the great work! I have a question about the forward_cond function under the RefOnlyNoisedUNet class. Why is it necessary to call it in the forward function? What does it achieve? Thank you!

https://github.com/SUDO-AI-3D/zero123plus/blob/main/diffusers-support/pipeline.py#L134

eliphatfs commented 5 months ago

It is the implementation of the Reference Attention mechanism. We forward UNet on the condition branch and record the KV matrices in the attention processor, then forward UNet on the denoising branch and concat the recorded KV matrices.

ZZWENG commented 5 months ago

thanks