RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one.

dvlab-research / FocalsConv

Focal Sparse Convolutional Networks for 3D Object Detection (CVPR 2022, Oral)

https://arxiv.org/abs/2204.12463

Apache License 2.0

371 stars 35 forks source link

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. #17

Closed zijianh4 closed 2 years ago

zijianh4 commented 2 years ago

Hi, Thanks for your great work! Currently, I am trying to train the model Voxel R-CNN (Car) + Focals Conv (multimodal) without road_plane. Therefore, I modify the kitti_dataset.yaml and voxel_rcnn_car_focal_multimodal.yaml from USE_ROAD_PLANE: True to USE_ROAD_PLANE: False. However, I encounter this RuntimeError, and don't know how to fix it.

Could you please help me? Thanks! :)

yukang2017 commented 2 years ago

I think you can make the program runable by adding find_unused_parameters=true in this line. https://github.com/dvlab-research/FocalsConv/blob/ff7568e0651886e7a9e0dc8f75b78f11224be4d7/OpenPCDet/tools/train.py#L142

However, I never test it without USE_ROAD_PLANE. Thus, I am not sure about the performance. You can try it.