got infinite loss when training modelnet40 dataset

Hello Christopher, I'm trying to evaluate your model on the synthetic ModelNet40 dataset. I wrote my own dataloader and trained your model with the default settings. But after a few steps, the loss becomes infinite. Can you give some suggestions about how to train your model on ModelNet40 dataset? (I've changed and tried different voxel sizes choose from 0.01 to 0.05, but did not work.)

A piece of logs during training can be found below. It seems the returned 'ws' variables from weighted_procrustes are all lower than the threshold (10 in your code), which means all input items have less than 10 valid correspondences pairs.

12/27 19:11:19 => loading weights for inlier model './FCGF_pretrained_3dmatch.pth' 12/27 19:11:19 => Loaded base model weights from './FCGF_pretrained_3dmatch.pth' 12/27 19:11:19 Inlier weight not found in './FCGF_pretrained_3dmatch.pth' /home/mmvc/anaconda3/envs/XL_py3_cuda10/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:449: UserWarning: To get the last learning rate computed by the scheduler, please use get_last_lr(). "please use get_last_lr().", UserWarning) 12/27 19:11:19 Epoch: 1, LR: [0.1] num valid_mask, weights.mean(), weights.max 6141 0.49184879660606384 0.8573861122131348 num_valid, ws 8 tensor([379.1353, 379.4299, 379.1117, 378.9486, 366.3976, 379.6558, 379.5316, 379.6907]) rot_error, trans_error, loss 1.0711798667907715 0.260419100522995 1.3315989971160889 12/27 19:11:23 Train Epoch: 1 [0/1230], Current Loss: 2.011e+00, Correspondence acc: 2.116e-02 , Precision: 0.0156, Recall: 0.0154, F1: 0.0155, TPR: 0.0154, TNR: 0.9790, BAcc: 0.4972 RTE: 2.604e-01, RRE: 6.137e+01, Succ rate: 2.500000e-01 Avg num valid: 8.000000e+00 Data time: 2.2289, Train time: 1.5519, NN search time: 1.772e-02, Total time: 3.7808 num valid_mask, weights.mean(), weights.max 6141 0.4818011224269867 0.8922387957572937 num_valid, ws 8 tensor([370.2599, 365.9893, 370.7393, 370.6736, 370.6388, 370.6003, 370.6378, 370.5956]) rot_error, trans_error, loss 0.7573173642158508 0.19806239008903503 0.9553797245025635 num valid_mask, weights.mean(), weights.max 6143 0.45103055238723755 0.9931040406227112 num_valid, ws 8 tensor([345.9790, 343.8760, 344.7417, 344.6624, 344.8690, 345.0934, 357.2584, 344.6322]) rot_error, trans_error, loss 0.7590370774269104 0.26072144508361816 1.0197584629058838 num valid_mask, weights.mean(), weights.max 6144 0.42308372259140015 0.9964439272880554 num_valid, ws 8 tensor([327.4283, 324.3058, 324.6727, 324.4388, 324.5334, 324.9588, 324.6275, 324.4608]) rot_error, trans_error, loss 1.3009799718856812 0.21060435473918915 1.5115842819213867 num valid_mask, weights.mean(), weights.max 6144 0.38447126746177673 0.9924339056015015 num_valid, ws 8 tensor([309.3771, 293.2216, 293.0517, 294.7009, 293.1519, 292.7238, 292.6430, 293.3217]) rot_error, trans_error, loss 0.544593334197998 0.2057737112045288 0.7503671646118164 num valid_mask, weights.mean(), weights.max 6143 0.34405946731567383 0.8312152028083801 num_valid, ws 8 tensor([265.8239, 263.1904, 265.2418, 266.2042, 263.8054, 263.6702, 263.0773, 262.8550]) rot_error, trans_error, loss 1.1339480876922607 0.24252642691135406 1.3764744997024536 num valid_mask, weights.mean(), weights.max 6144 0.2863656282424927 0.9999984502792358 num_valid, ws 8 tensor([222.1238, 218.1181, 215.6836, 226.8676, 217.5520, 217.8743, 220.7595, 220.4514]) rot_error, trans_error, loss 1.0707015991210938 0.27141687273979187 1.3421183824539185 num valid_mask, weights.mean(), weights.max 6144 0.21661357581615448 0.34146520495414734 num_valid, ws 8 tensor([173.3432, 162.5075, 164.1453, 168.3337, 164.7490, 168.9405, 158.8151, 170.0394]) rot_error, trans_error, loss 0.8970396518707275 0.30696189403533936 1.2040014266967773 num valid_mask, weights.mean(), weights.max 6138 0.22650477290153503 0.9914805889129639 num_valid, ws 8 tensor([171.5588, 173.3483, 179.2511, 173.2488, 172.3625, 172.5608, 172.9400, 176.2171]) rot_error, trans_error, loss 1.0415538549423218 0.34417128562927246 1.3857251405715942 num valid_mask, weights.mean(), weights.max 6116 0.17247292399406433 0.35712677240371704 num_valid, ws 8 tensor([132.9017, 130.7651, 128.1572, 133.2996, 131.4898, 134.0894, 132.3888, 135.9132]) rot_error, trans_error, loss 0.9503939151763916 0.22698664665222168 1.1773805618286133 num valid_mask, weights.mean(), weights.max 6064 0.1356555074453354 0.7415195107460022 num_valid, ws 8 tensor([104.9554, 104.2914, 102.3165, 103.3207, 103.8977, 103.7889, 103.3343, 105.0705]) rot_error, trans_error, loss 0.5307180881500244 0.16665557026863098 0.697373628616333 num valid_mask, weights.mean(), weights.max 6048 0.12273039668798447 0.5408609509468079 num_valid, ws 8 tensor([92.7914, 91.0353, 90.9816, 92.6906, 98.6065, 94.2565, 96.6829, 94.6045]) rot_error, trans_error, loss 0.7709248661994934 0.1336054652929306 0.9045303463935852 num valid_mask, weights.mean(), weights.max 5288 0.07958836853504181 0.17408770322799683 num_valid, ws 8 tensor([59.2375, 49.9505, 57.0326, 52.8035, 60.2806, 60.2326, 60.3720, 57.5639]) rot_error, trans_error, loss 1.1103489398956299 0.25814875960350037 1.3684978485107422 num valid_mask, weights.mean(), weights.max 3978 0.06213882565498352 0.18988150358200073 num_valid, ws 8 tensor([32.8604, 31.9178, 39.7851, 39.5970, 37.5519, 40.9863, 38.9679, 40.8111]) rot_error, trans_error, loss 1.3329100608825684 0.35916927456855774 1.6920795440673828 num valid_mask, weights.mean(), weights.max 3824 0.05560467392206192 0.16689369082450867 num_valid, ws 8 tensor([34.9690, 44.5694, 32.6451, 32.3816, 34.3029, 31.2937, 31.9045, 32.9062]) rot_error, trans_error, loss 1.1255593299865723 0.261192262172699 1.386751413345337 num valid_mask, weights.mean(), weights.max 2160 0.04612640663981438 0.19192813336849213 num_valid, ws 8 tensor([21.0208, 19.3035, 14.4086, 24.3450, 20.2616, 22.0708, 29.0271, 16.9002]) rot_error, trans_error, loss 0.9262062311172485 0.3681674003601074 1.2943737506866455 num valid_mask, weights.mean(), weights.max 4355 0.05530725419521332 0.10722033679485321 num_valid, ws 8 tensor([34.5013, 41.0727, 37.0651, 35.8570, 38.5333, 39.4196, 36.3496, 38.5415]) rot_error, trans_error, loss 0.977398157119751 0.1942673623561859 1.1716655492782593 num valid_mask, weights.mean(), weights.max 1208 0.03593476861715317 0.09210973232984543 num_valid, ws 5 tensor([11.2543, 8.6302, 11.9625, 9.4134, 14.0098, 13.0315, 11.4335, 5.5197]) rot_error, trans_error, loss 1.2372373342514038 0.310983270406723 1.81670343875885 num valid_mask, weights.mean(), weights.max 2057 0.040355708450078964 0.10537022352218628 num_valid, ws 8 tensor([17.9894, 16.3948, 18.6110, 18.1689, 11.5592, 15.3969, 13.3000, 15.5626]) rot_error, trans_error, loss 0.769038736820221 0.1769949048757553 0.9460336565971375 num valid_mask, weights.mean(), weights.max 1714 0.036536455154418945 0.0762948989868164 num_valid, ws 7 tensor([17.7795, 12.5434, 10.8226, 14.2316, 14.7396, 10.7642, 7.0441, 12.2532]) rot_error, trans_error, loss 0.6872734427452087 0.21808893978595734 0.9117898941040039 num valid_mask, weights.mean(), weights.max 572 0.028709784150123596 0.08586414158344269 num_valid, ws 0 tensor([2.6134, 7.0487, 4.7059, 3.5765, 4.7559, 5.8939, 6.2377, 0.6551]) rot_error, trans_error, loss 1.3239400386810303 0.4019933044910431 nan 12/27 19:12:02 Loss is infinite, abort num valid_mask, weights.mean(), weights.max 680 0.03068387135863304 0.09823115170001984 num_valid, ws 0 tensor([4.6802, 3.1837, 6.2774, 6.7470, 5.4355, 6.1506, 3.3652, 6.2467]) rot_error, trans_error, loss 1.7035499811172485 0.46357956528663635 nan 12/27 19:12:04 Loss is infinite, abort num valid_mask, weights.mean(), weights.max 1567 0.03716740012168884 0.09059126675128937 num_valid, ws 5 tensor([18.5755, 16.7832, 11.3389, 9.6333, 11.9852, 9.1712, 5.9884, 11.1698]) rot_error, trans_error, loss 0.9750125408172607 0.32420605421066284 1.2242933511734009 num valid_mask, weights.mean(), weights.max 1137 0.03218415006995201 0.09962588548660278 num_valid, ws 3 tensor([12.0679, 8.3144, 9.6743, 9.2133, 3.5999, 8.8996, 11.8518, 10.5215]) rot_error, trans_error, loss 0.5337458252906799 0.19239680469036102 0.6585435271263123 num valid_mask, weights.mean(), weights.max 966 0.029521774500608444 0.10332392156124115 num_valid, ws 0 tensor([8.4000, 8.9959, 6.1997, 8.6553, 9.4742, 7.0023, 8.5738, 4.6189]) rot_error, trans_error, loss 1.0506246089935303 0.3039046823978424 nan 12/27 19:12:10 Loss is infinite, abort num valid_mask, weights.mean(), weights.max 1074 0.02992144785821438 0.10137491673231125 num_valid, ws 2 tensor([12.6404, 8.2678, 3.2425, 12.0476, 6.9744, 5.5812, 9.3145, 8.7726]) rot_error, trans_error, loss 1.611760139465332 0.4179706275463104 1.2622950077056885 num valid_mask, weights.mean(), weights.max 1543 0.034905172884464264 0.09911085665225983 num_valid, ws 5 tensor([10.5303, 9.5016, 17.0358, 9.5487, 14.0452, 16.5256, 8.4574, 10.9382]) rot_error, trans_error, loss 1.512508749961853 0.3802349269390106 2.0282459259033203 num valid_mask, weights.mean(), weights.max 1506 0.03240210562944412 0.1912095695734024 num_valid, ws 4 tensor([10.4049, 8.4145, 12.4113, 8.7133, 10.1371, 16.9338, 7.0592, 9.6483]) rot_error, trans_error, loss 0.7196844816207886 0.161781445145607 0.6110647916793823 num valid_mask, weights.mean(), weights.max 831 0.02881685458123684 0.10011401772499084 num_valid, ws 0 tensor([7.1511, 2.9459, 9.2325, 5.5985, 2.2618, 4.2966, 4.5117, 9.9505]) rot_error, trans_error, loss 1.474617600440979 0.3669683635234833 nan 12/27 19:12:18 Loss is infinite, abort num valid_mask, weights.mean(), weights.max 449 0.026938147842884064 0.09409534186124802 num_valid, ws 0 tensor([5.2363, 2.1232, 1.9240, 1.4735, 2.0403, 1.9763, 6.4409, 3.1939]) rot_error, trans_error, loss 1.2611165046691895 0.29535406827926636 nan 12/27 19:12:20 Loss is infinite, abort

chrischoy / DeepGlobalRegistration

got infinite loss when training modelnet40 dataset #20