search

fundamentalvision / BEVFormer

[ECCV 2022] This is the official implementation of BEVFormer, a camera-only framework for autonomous driving perception, e.g., 3D object detection and semantic map segmentation.
https://arxiv.org/abs/2203.17270
Apache License 2.0
3.39k stars 550 forks source link

ms_deform_attn_backward error #256

Open Mandylove1993 opened 6 months ago

Mandylove1993 commented 6 months ago

训练模型时，报错“RuntimeError: ms_deform_attn_impl_forward: implementation for device cuda:0 not found.”，怎么办 How can I do for it ??Thanks,

Traceback (most recent call last): File "./tools/train.py", line 259, in main() File "./tools/train.py", line 248, in main custom_train_model( File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/apis/train.py", line 27, in custom_train_model custom_train_detector( File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/apis/mmdet_train.py", line 199, in custom_train_detector runner.run(data_loaders, cfg.workflow) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run epoch_runner(data_loaders[i], kwargs) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train self.run_iter(data_batch, train_mode=True, kwargs) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 29, in run_iter outputs = self.model.train_step(data_batch, self.optimizer, File "/opt/conda/lib/python3.8/site-packages/mmcv/parallel/distributed.py", line 52, in train_step output = self.module.train_step(inputs[0], kwargs[0]) File "/opt/conda/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 248, in train_step losses = self(data) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, kwargs) File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/detectors/bevformer_occ.py", line 154, in forward return self.forward_train(kwargs) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func return old_func(*args, kwargs) File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/detectors/bevformer_occ.py", line 225, in forward_train prev_bev = self.obtain_history_bev(prev_img, prev_img_metas) File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/detectors/bevformer_occ.py", line 174, in obtain_history_bev prev_bev = self.pts_bbox_head( File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func return old_func(args, kwargs) File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/dense_heads/bevformer_occ_head.py", line 124, in forward return self.transformer.get_bev_features( File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func return old_func(*args, kwargs) File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/modules/transformer_occ.py", line 261, in get_bev_features bev_embed = self.encoder( File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 98, in new_func return old_func(args, kwargs) File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/modules/encoder.py", line 217, in forward output = layer( File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, kwargs) File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/modules/encoder.py", line 362, in forward query = self.attentions[attn_index]( File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/modules/temporal_self_attention.py", line 247, in forward output = MultiScaleDeformableAttnFunction.apply( File "/opt/conda/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 94, in decorate_fwd return fwd(args, kwargs) File "/ssdb/zft/39_server/occ/code/CVPR2023-3D-Occupancy-Prediction/projects/mmdet3d_plugin/bevformer/modules/multi_scale_deformable_attn_function.py", line 118, in forward output = ext_module.ms_deform_attn_forward( RuntimeError: ms_deform_attn_impl_forward: implementation for device cuda:1 not found.

lix19937 commented 6 months ago

This step

e. Install mmdet3d from source code.

git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout v0.17.1 # Other versions may not be compatible.
python setup.py install

should install right.

Shwetali-Wadekar commented 2 months ago

I am facing the same error and could not resolve it. @Mandylove1993 did you resolve this issue?