xinge008 / Cylinder3D

Rank 1st in the leaderboard of SemanticKITTI semantic segmentation (both single-scan and multi-scan) (Nov. 2020) (CVPR2021 Oral)
Apache License 2.0
859 stars 180 forks source link

Trying to do binary classification with BCEloss(), but I cannot make the model converge #124

Closed JohanBergius closed 2 years ago

JohanBergius commented 2 years ago

Hi, I'm trying to do the binary classification of outliers with your code, but the loss gets stuck at 0.688 (not converged). Any suggestions?

29%|██▉ | 192/658 [02:39<06:15, 1.24it/s]loss tensor(0.6868, device='cuda:0', grad_fn=) 30%|██▉ | 197/658 [02:44<06:07, 1.26it/s]loss tensor(0.6876, device='cuda:0', grad_fn=) 31%|███ | 202/658 [02:48<06:13, 1.22it/s]loss tensor(0.6883, device='cuda:0', grad_fn=) 31%|███▏ | 207/658 [02:52<06:08, 1.22it/s]loss tensor(0.6887, device='cuda:0', grad_fn=) 32%|███▏ | 212/658 [02:56<06:00, 1.24it/s]loss tensor(0.6896, device='cuda:0', grad_fn=) 33%|███▎ | 217/658 [03:00<05:44, 1.28it/s]loss tensor(0.6870, device='cuda:0', grad_fn=) 34%|███▎ | 222/658 [03:03<05:35, 1.30it/s]loss tensor(0.6880, device='cuda:0', grad_fn=) 34%|███▍ | 227/658 [03:08<05:45, 1.25it/s]loss tensor(0.6876, device='cuda:0', grad_fn=) 35%|███▌ | 232/658 [03:12<05:45, 1.23it/s]loss tensor(0.6872, device='cuda:0', grad_fn=) 36%|███▌ | 237/658 [03:16<05:41, 1.23it/s]loss tensor(0.6876, device='cuda:0', grad_fn=) 37%|███▋ | 242/658 [03:20<05:43, 1.21it/s]loss tensor(0.6881, device='cuda:0', grad_fn=) 38%|███▊ | 247/658 [03:24<05:37, 1.22it/s]loss tensor(0.6912, device='cuda:0', grad_fn=) 38%|███▊ | 252/658 [03:28<05:30, 1.23it/s]loss tensor(0.6889, device='cuda:0', grad_fn=) 39%|███▉ | 257/658 [03:32<05:21, 1.25it/s]loss tensor(0.6868, device='cuda:0', grad_fn=) 40%|███▉ | 262/658 [03:36<05:11, 1.27it/s]loss tensor(0.6873, device='cuda:0', grad_fn=) 41%|████ | 267/658 [03:40<05:18, 1.23it/s]loss tensor(0.6883, device='cuda:0', grad_fn=) 41%|████▏ | 272/658 [03:44<05:07, 1.25it/s]loss tensor(0.6872, device='cuda:0', grad_fn=) 42%|████▏ | 277/658 [03:48<05:13, 1.22it/s]loss tensor(0.6869, device='cuda:0', grad_fn=) 43%|████▎ | 282/658 [03:52<05:04, 1.23it/s]loss tensor(0.6885, device='cuda:0', grad_fn=) 44%|████▎ | 287/658 [03:56<04:54, 1.26it/s]loss tensor(0.6890, device='cuda:0', grad_fn=) 44%|████▍ | 292/658 [04:00<04:50, 1.26it/s]loss tensor(0.6877, device='cuda:0', grad_fn=) 45%|████▍ | 293/658 [04:01<04:50, 1.26it/s]

JohanBergius commented 2 years ago

I found a thing that could be related to the problem:

My labels only include zeros and ones. However, the label tensor has"10952165" samples corresponding to the "ignore_label" attribute (I have used ignore_label: 2 here). Why is this the case? (I don't want to change dataset_semantickitti.py unless I have to)

OUTPUT:

assert False, torch.bincount(torch.flatten(point_label_tensor))

AssertionError: tensor([ 70595, 36440, 10952165], device='cuda:0')