Error when run with crm=true

[sekkit]

Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is None and using 5 heads. Setting up MemoryEfficientCrossAttention. Query dim is 320, context_dim is 1024 and using 5 heads. making attention of type 'vanilla-xformers' with 512 in_channels building MemoryEfficientAttnBlock with 512 in_channels... Working with z of shape (1, 4, 32, 32) = 4096 dimensions. making attention of type 'vanilla-xformers' with 512 in_channels building MemoryEfficientAttnBlock with 512 in_channels... --- using zero snr--- Traceback (most recent call last): File "/home/work/dev/LucidFusion/demo.py", line 128, in main() File "/home/work/dev/LucidFusion/demo.py", line 120, in main pipeline = setup_crm_diffusion() File "/home/work/dev/LucidFusion/data/diffusion_data.py", line 40, in setup_crm_diffusion pipeline = TwoStagePipeline( File "/home/work/dev/LucidFusion/CRM/pipelines.py", line 42, in init self.stage1_sampler = get_obj_from_str(stage1_sampler_config.target)( File "/home/work/dev/LucidFusion/CRM/libs/sample.py", line 32, in init uc = model.get_learned_conditioning([neg_texts]).to(device) File "/home/work/dev/LucidFusion/CRM/imagedream/ldm/interface.py", line 185, in get_learned_conditioning return self.clip_model(prompts) File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, *kwargs) File "/home/work/dev/LucidFusion/CRM/imagedream/ldm/modules/encoders/modules.py", line 218, in forward z = self.encode_with_transformer(tokens.to(self.device)) File "/home/work/dev/LucidFusion/CRM/imagedream/ldm/modules/encoders/modules.py", line 282, in encode_with_transformer x = self.text_transformer_forward(x, attn_mask=self.model.attn_mask) File "/home/work/dev/LucidFusion/CRM/imagedream/ldm/modules/encoders/modules.py", line 297, in text_transformer_forward x = r(x, attn_mask=attn_mask) File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, kwargs) File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/open_clip/transformer.py", line 263, in forward x = q_x + self.ls_1(self.attention(q_x=self.ln_1(q_x), k_x=k_x, v_x=v_x, attn_mask=attn_mask)) File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/open_clip/transformer.py", line 250, in attention return self.attn( File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(args, kwargs) File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/torch/nn/modules/activation.py", line 1266, in forward attn_output, attn_output_weights = F.multi_head_attention_forward( File "/home/sekkit/miniconda3/envs/DreamMat/lib/python3.10/site-packages/torch/nn/functional.py", line 5382, in multi_head_attention_forward raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.") RuntimeError: The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (1, 1).

[heye0507]

hi seems to me it is the open-clip version issue.

can you try:

pip install open-clip-torch==2.24.0

Let me know if it works. :)