RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one

dvlab-research / FocalsConv

Focal Sparse Convolutional Networks for 3D Object Detection (CVPR 2022, Oral)

https://arxiv.org/abs/2204.12463

Apache License 2.0

371 stars 35 forks source link

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one #36

Open saeid114 opened 5 months ago

saeid114 commented 5 months ago

hi @yukang2017 @yanwei-li I have already tried this (find_unused_parameters=True), but the same error persists. Do you have any solutions? if dist_train: model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()], find_unused_parameters=True)

Screenshot 2024-06-08 at 1 18 43 AM