error when training on 1 gpu

CVMI-Lab / PLA

(CVPR 2023) PLA: Language-Driven Open-Vocabulary 3D Scene Understanding & (CVPR2024) RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding

Apache License 2.0

262 stars 11 forks source link

error when training on 1 gpu #16

Closed Pixie8888 closed 1 year ago

Pixie8888 commented 1 year ago

Hi, Thanks for releasing code! When I training S3DIS on one gpu, I got an error below. It seems that not all modules are participated during training? How to solve it?

Dingry commented 1 year ago

This issue is likely caused by the caption_logit_scale not being utilized in the loss computation. To verify if the caption loss is zero, you can set find_unused_parameters=True. Note that this problem is more likely to occur when working with fewer GPUs (fewer scenes in a batch) due to missing 2D image data in certain scenes in S3DIS.

Pixie8888 commented 1 year ago

Thanks! setting find_unused_parameters=True works.