A question about reconstruction.py

[songqianyi01]

During the image reconstruction phase, I encountered such an error, have you ever encountered it？

Traceback (most recent call last): File "/mnt/sda/songzengyu/code/MindDiffuser-main/Image reconstruction/Reconstruction_new.py", line 417, in main() File "/mnt/sda/songzengyu/code/MindDiffuser-main/Image reconstruction/Reconstruction_new.py", line 392, in main samples = sampler.decode(z_enc, c, 35, unconditional_guidance_scale=5.0, unconditional_conditioning=uc, ) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/models/diffusion/ddim.py", line 238, in decode xdec, = self.p_sample_ddim(x_dec, cond, ts, index=index, use_original_steps=use_original_steps, File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/models/diffusion/ddim.py", line 177, in p_sample_ddim e_t_uncond, e_t = self.model.apply_model(x_in, t_in, c_in).chunk(2) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/models/diffusion/ddpm.py", line 987, in apply_model x_recon = self.model(x_noisy, t, cond) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/models/diffusion/ddpm.py", line 1410, in forward out = self.diffusion_model(x, t, context=cc) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/modules/diffusionmodules/openaimodel.py", line 732, in forward h = module(h, emb, context) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, *kwargs) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/modules/diffusionmodules/openaimodel.py", line 85, in forward x = layer(x, context) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/modules/attention.py", line 258, in forward x = block(x, context=context) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/modules/attention.py", line 209, in forward return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/modules/diffusionmodules/util.py", line 114, in checkpoint return CheckpointFunction.apply(func, len(inputs), args) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/autograd/function.py", line 553, in apply return super().apply(args, kwargs) # type: ignore[misc] File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/modules/diffusionmodules/util.py", line 127, in forward output_tensors = ctx.run_function(ctx.input_tensors) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/modules/attention.py", line 213, in _forward x = self.attn2(self.norm2(x), context=context) + x File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, *kwargs) File "/mnt/sda/songzengyu/code/stable-diffusion-main/ldm/modules/attention.py", line 180, in forward sim = einsum('b i d, b j d -> b i j', q, k) self.scale File "/home/songzengyu/.conda/envs/ldm/lib/python3.8/site-packages/torch/functional.py", line 380, in einsum return _VF.einsum(equation, operands) # type: ignore[attr-defined] RuntimeError: einsum(): subscript b has size 48 for operand 1 which does not broadcast with previously seen size 80

[ReedOnePeck]

I have not encountered this issue before. From the error message you provided, it seems that there is a problem with the input feature dimensions on line 392 of the Reconstruction_new.py file. What are the dimensions of the input 'z_enc' and 'c'?

[songqianyi01]

Their dimension is here,zenc[5,4,64,64],c[5,15,768]。Because I was told that cuda memory was insufficient when 982 test data were loaded, I chose 5 sets of data

[ReedOnePeck]

Our model reconstructs images one by one and cannot process in parallel; therefore, the dimensions of zenc and c should be [1, 4, 64, 64] and [1, 15, 768] respectively. Try reconstructing just one image to see if the error still occurs.

[songqianyi01]

Ok, I see. Thank you for your reply. I'll have a try