search

TRI-ML / vidar

Other
547 stars 65 forks source link

DepthFormer fails on dataset without 'pose' information #15

Open iariav opened 2 years ago

iariav commented 2 years ago

Hi, when training a DepthFormerModel, in the process_stereo function

https://github.com/TRI-ML/vidar/blob/a306e23cc47ae05a0c35b0bf8acf7112c87c049c/vidar/arch/models/depth/DepthFormerModel.py#L64

it fails if there is no 'pose' in the batch. is it designed to work only with datasets with pose information (training on custom dataset)? thanks

VitorGuizilini-TRI commented 2 years ago

DepthFormer as discussed in the paper doesn't require pose information, however it was designed to also work in the generalized stereo setting, using multiple cameras at each time step. Seems like the model requires this information, I'll try to modify it soon to support temporal-only, but in the meantime you can just comment out all parts related to stereo.

CydKnocking commented 2 years ago

Hi, when training a DepthFormerModel, in the process_stereo function

https://github.com/TRI-ML/vidar/blob/a306e23cc47ae05a0c35b0bf8acf7112c87c049c/vidar/arch/models/depth/DepthFormerModel.py#L64

it fails if there is no 'pose' in the batch. is it designed to work only with datasets with pose information (training on custom dataset)? thanks

Hi iarav, I met the same problem with you. And I tried replacing line 64~67 with the followings:

        if 'pose' in batch:
            for key in batch['pose'].keys():
                if not is_str(key) and key != 0:
                    new_intrinsics[key] = batch['intrinsics'][0]
        else:
            new_intrinsics[-1] = batch['intrinsics'][0]
            new_intrinsics[1] = batch['intrinsics'][0]
        batch['intrinsics'] = new_intrinsics

and that works well for me. Hopefully this will help you!

NorthSummer commented 1 year ago

Hi, when training a DepthFormerModel, in the process_stereo function https://github.com/TRI-ML/vidar/blob/a306e23cc47ae05a0c35b0bf8acf7112c87c049c/vidar/arch/models/depth/DepthFormerModel.py#L64

it fails if there is no 'pose' in the batch. is it designed to work only with datasets with pose information (training on custom dataset)? thanks

Hi iarav, I met the same problem with you. And I tried replacing line 64~67 with the followings:

        if 'pose' in batch:
            for key in batch['pose'].keys():
                if not is_str(key) and key != 0:
                    new_intrinsics[key] = batch['intrinsics'][0]
        else:
            new_intrinsics[-1] = batch['intrinsics'][0]
            new_intrinsics[1] = batch['intrinsics'][0]
        batch['intrinsics'] = new_intrinsics

and that works well for me. Hopefully this will help you!

Hi, don't know if you are still there; I replace the code as you have mentioned above, the "pose" bug is solved, however a new bug jump out as follows: Traceback (most recent call last): File "scripts/run.py", line 25, in fire.Fire(train) File "/root/miniconda3/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/root/miniconda3/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/root/miniconda3/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace component = fn(*varargs, *kwargs) File "scripts/run.py", line 21, in train trainer.learn(wrapper) File "/root/autodl-tmp/code/vidar/vidar/core/trainer.py", line 354, in learn self.train(dataloaders['train'], optimizers, schedulers, wrapper, scaler=scaler, File "/root/autodl-tmp/code/vidar/vidar/core/trainer.py", line 401, in train output = wrapper.training_step(batch, epoch=self.current_epoch) File "/root/autodl-tmp/code/vidar/vidar/core/wrapper.py", line 176, in training_step output = self.run_arch(batch, epoch=epoch, flip=flip_lr, unflip=False) File "/root/autodl-tmp/code/vidar/vidar/core/wrapper.py", line 164, in run_arch output = self.arch(batch, epoch=epoch) File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, **kwargs) File "/root/autodl-tmp/code/vidar/vidar/arch/models/depth/DepthFormerModel.py", line 468, in forward supervision_output = self.losses['supervision'](depth_pred, depth_gt) File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/container.py", line 313, in getitem return self._modules[key] KeyError: 'supervision'

Have you ever met this? thanks