Open Assoillyng opened 1 year ago
Turns out I was missing something: self.diffusionmodel is of type openaimodel.UNetModel, and does take context and y. Though I'm still having a lot of trouble getting unclip to run, the implementation seems to be incomplete.
I second this. The implementation seems to be incomplete.
I might be misunderstanding the implementation, but from what I can trace, it seems like context and y aren't being utilized in the p_losses method. Could someone provide some clarity on this?
Expected Behavior: I expected context and y to be used within the p_losses method, given that they're passed as arguments in the forward method of LatentDiffusion.
Actual Behavior: context and y appear to be discarded and not used in p_losses.
Context: In v2-1-stable-unclip-h-inference.yaml we can see that the target is ImageEmbeddingConditionedLatentDiffusion and the conditioning key is crossattn-adm.
Further, we can see this code in ddpm.DiffusionWrapper:
And we see that ImageEmbeddingConditionedLatentDiffusion inherits from LatentDiffusion and has no foward method of its own, so must use the forward method from LatentDiffusion:
This passes the keyword arguments (context and y) through to p_losses, which promptly throws them away as it doesn't accept **kwargs:
So we have no embeddings for the images (c_adm) being used anywhere and no cross attention data (cc) being used anywhere.
Am I missing something here or is this a bug?