search

xinge008 / Cylinder3D

Rank 1st in the leaderboard of SemanticKITTI semantic segmentation (both single-scan and multi-scan) (Nov. 2020) (CVPR2021 Oral)
Apache License 2.0
858 stars 180 forks source link

Stuck when start training #143

Closed mc171819 closed 2 years ago

mc171819 commented 2 years ago

Hi, I tried to run your training code, however, I cannot start training successfully. I have stucked here for a long time. Do you have advice to start training? image @xinge008

mc171819 commented 2 years ago

The CPU occupation is high now but gpu occupation is low. Other project work fine on gpu so it is not my gpu problem. This is the output of nvidia-smi image

mc171819 commented 2 years ago

after waiting for about half an hour, it start training and all things seems normal. It seems epoch 0 iter0 is not on gpu training. It's an weird problem. image

mc171819 commented 2 years ago

The problem is the validation of iter 0 ep 0 before training, add global_iter>0 on validation solute it.