search

maybeLx / MVSFormerPlusPlus

Codes of MVSFormer++: Revealing the Devil in Transformer’s Details for Multi-View Stereo (ICLR2024)
Apache License 2.0
178 stars 6 forks source link

finetune using our custom data #22

Open mawenjie1111 opened 4 months ago

mawenjie1111 commented 4 months ago

we use our data to finetune with pretrained dtu model, our data used the same format as DTU. but we get some error when training

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/root/miniconda3/envs/mvsformer/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 74, in _wrap fn(i, args) File "/root/autodl-tmp/code/MVSFormerPlusPlus/train.py", line 206, in main trainer.train() File "/root/autodl-tmp/code/MVSFormerPlusPlus/base/base_trainer.py", line 79, in train result = self._train_epoch(epoch) File "/root/autodl-tmp/code/MVSFormerPlusPlus/trainer/mvsformer_trainer.py", line 176, in _train_epoch self.scaler.step(self.optimizer) File "/root/miniconda3/envs/mvsformer/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 416, in step retval = self._maybe_opt_step(optimizer, optimizer_state, args, kwargs) File "/root/miniconda3/envs/mvsformer/lib/python3.10/site-packages/torch/cuda/amp/grad_scaler.py", line 315, in _maybe_opt_step retval = optimizer.step(*args, *kwargs) File "/root/miniconda3/envs/mvsformer/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper return wrapped(args, kwargs) File "/root/miniconda3/envs/mvsformer/lib/python3.10/site-packages/torch/optim/optimizer.py", line 373, in wrapper out = func(*args, *kwargs) File "/root/miniconda3/envs/mvsformer/lib/python3.10/site-packages/torch/optim/optimizer.py", line 76, in _use_grad ret = func(self, args, **kwargs) File "/root/miniconda3/envs/mvsformer/lib/python3.10/site-packages/torch/optim/adamw.py", line 184, in step adamw( File "/root/miniconda3/envs/mvsformer/lib/python3.10/site-packages/torch/optim/adamw.py", line 335, in adamw func( File "/root/miniconda3/envs/mvsformer/lib/python3.10/site-packages/torch/optim/adamw.py", line 540, in _multi_tensor_adamw torch._foreachlerp(device_exp_avgs, device_grads, 1 - beta1) RuntimeError: The size of tensor a (64) must match the size of tensor b (768) at non-singleton dimension 0

Do you have any suggestions that can help us solve this problem? Or how to finetune data similar to DTU

maybeLx commented 3 months ago

You can use single GPU to check which line raises error