search

PJLab-ADG / DetZero

[ICCV 2023] DetZero: Rethinking Offboard 3D Object Detection with Long-term Sequential Point Clouds
https://superkoma.github.io/detzero-page/
Apache License 2.0
334 stars 32 forks source link

"RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input." #17

Open rockywind opened 1 year ago

rockywind commented 1 year ago

Hi, thank you for sharing the great work! I met the error of below when I run the script below.

python test.py --cfg_file cfgs/det_model_cfgs/centerpoint_1sweep_my.yaml \
--ckpt /SHFP12/xiaoquan.wang/01_bev/DetZero/detection/output/det_model_cfgs/centerpoint_1sweep_my/default/ckpt/checkpoint_epoch_30.pth \
--extra_tag centerpoint_1sweep_my_tta \
--set DATA_CONFIG.TTA True

The error is below.

023-09-26 11:50:50,402   INFO  *************** EPOCH 30 EVALUATION *****************
eval:   0%|                                                                             | 0/1250 [00:00<?, ?it/s]Traceback (most recent call last):
  File "test.py", line 205, in <module>
    main()
  File "test.py", line 201, in main
    eval_single_ckpt(model, test_loader, args, eval_output_dir, logger, epoch_id, dist_test=dist_test)
  File "test.py", line 66, in eval_single_ckpt
    eval_utils.eval_one_epoch(
  File "/defaultShare/SHFP12/xiaoquan.wang/01_bev/DetZero/detection/tools/eval_utils.py", line 73, in eval_one_epoch
    result = model(batch_dict)
  File "/opt/conda/envs/detzero/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/defaultShare/SHFP12/xiaoquan.wang/01_bev/DetZero/detection/detzero_det/models/centerpoint.py", line 29, in forward
    batch_dict = cur_module(batch_dict)
  File "/opt/conda/envs/detzero/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/defaultShare/SHFP12/xiaoquan.wang/01_bev/DetZero/detection/detzero_det/models/centerpoint_modules/backbone2d.py", line 101, in forward
    x = self.blocks[i](x)
  File "/opt/conda/envs/detzero/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/envs/detzero/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forward
    input = module(input)
  File "/opt/conda/envs/detzero/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/envs/detzero/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
    return F.batch_norm(
  File "/opt/conda/envs/detzero/lib/python3.8/site-packages/torch/nn/functional.py", line 2282, in batch_norm
    return torch.batch_norm(
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

But when I remove the --set DATA_CONFIG.TTA True , it run successfully.

SampWEI commented 2 months ago

I met same error in refining and found solutions in CSDN. https://blog.csdn.net/jiaoyangwm/article/details/135845029 It says that your batch_size is too large.