Error during training

[yejr0229]

Hi,I meet some error in training the geometry,seems like is due to the down sample,here is the detail:

5%|█████▌ | 99/2001 [00:14<02:45, 11.51it/s]Initialize SDF; it: 100 10%|███████████ | 199/2001 [00:24<02:17, 13.15it/s]Initialize SDF; it: 200 15%|████████████████▌ | 299/2001 [00:33<02:19, 12.20it/s]Initialize SDF; it: 300 20%|██████████████████████▏ | 400/2001 [00:49<03:18, 8.07it/s] Traceback (most recent call last): File "/home/yejr/AIGC/gala-main/train.py", line 1313, in geometry, mat = optimize_mesh(glctx, geometry, mat, lgt, dataset_train, dataset_validate, File "/home/yejr/AIGC/gala-main/train.py", line 1043, in optimize_mesh sds_loss,img_loss, reg_loss = model(target, it, if_normal, if_pretrain= if_pretrain, scene_and_vertices=scene_and_vertices) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, kwargs) File "/home/yejr/AIGC/gala-main/train.py", line 775, in forward return self.geometry.tick(self.glctx, target, self.light, self.material, it , if_normal, self.guidance, File "/home/yejr/AIGC/gala-main/geometry/dmtet.py", line 1331, in tick sds_loss += self.apply_SDS(buffers=buffers_sds_hum, text_embeddings=text_embeddings, guidance=guidance, iteration=iteration) self.w_sds
File "/home/yejr/AIGC/gala-main/geometry/dmtet.py", line 837, in apply_SDS noise_pred = guidance(latents, noise, t, text_embeddings, control_images=control_images) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(input, kwargs) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, kwargs) File "/home/yejr/AIGC/gala-main/utils/sd.py", line 389, in forward down_block_res_samples, mid_block_res_sample = self.controlnet( File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, *kwargs) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/diffusers/models/controlnet.py", line 777, in forward sample, res_samples = downsample_block( File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(input, kwargs) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/diffusers/models/unet_2d_blocks.py", line 1086, in forward hidden_states = attn( File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, kwargs) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/diffusers/models/transformer_2d.py", line 315, in forward hidden_states = block( File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, *kwargs) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/diffusers/models/attention.py", line 218, in forward attn_output = self.attn2( File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(input, kwargs) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 417, in forward return self.processor( File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/diffusers/models/attention_processor.py", line 578, in call key = attn.to_k(encoder_hidden_states, scale=scale) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/diffusers/models/lora.py", line 224, in forward out = super().forward(hidden_states) File "/media/data4/yejr/conda_env/gala/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 shapes cannot be multiplied (308x1024 and 768x320)