search

V2AI / Det3D

World's first general purpose 3D object detection codebse.
https://arxiv.org/abs/1908.09492
Apache License 2.0
1.49k stars 298 forks source link

Trying to train cbfgs. All values are NaN. #43

Closed chowkamlee81 closed 4 years ago

chowkamlee81 commented 4 years ago

Kindly help all values are naN . Iam using single GPU

2020-01-07 17:22:53,040 - INFO - task : ['car'], loss: nan, cls_pos_loss: nan, cls_neg_loss: nan, dir_loss_reduced: nan, cls_loss_reduced: nan, loc_loss_reduced: nan, loc_loss_elem: ['nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan'], num_pos: 26.6600, num_neg: 31688.8400 2020-01-07 17:22:53,040 - INFO - task : ['truck', 'construction_vehicle'], loss: nan, cls_pos_loss: nan, cls_neg_loss: nan, dir_loss_reduced: nan, cls_loss_reduced: nan, loc_loss_reduced: nan, loc_loss_elem: ['nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan'], num_pos: 40.4800, num_neg: 63408.1400 2020-01-07 17:22:53,040 - INFO - task : ['bus', 'trailer'], loss: nan, cls_pos_loss: nan, cls_neg_loss: nan, dir_loss_reduced: nan, cls_loss_reduced: nan, loc_loss_reduced: nan, loc_loss_elem: ['nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan'], num_pos: 58.1800, num_neg: 63362.3000 2020-01-07 17:22:53,040 - INFO - task : ['barrier'], loss: nan, cls_pos_loss: nan, cls_neg_loss: nan, dir_loss_reduced: nan, cls_loss_reduced: nan, loc_loss_reduced: nan, loc_loss_elem: ['nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan'], num_pos: 7.8600, num_neg: 31742.0200 2020-01-07 17:22:53,040 - INFO - task : ['motorcycle', 'bicycle'], loss: nan, cls_pos_loss: nan, cls_neg_loss: nan, dir_loss_reduced: nan, cls_loss_reduced: nan, loc_loss_reduced: nan, loc_loss_elem: ['nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan'], num_pos: 11.8800, num_neg: 63486.6800 2020-01-07 17:22:53,040 - INFO - task : ['pedestrian', 'traffic_cone'], loss: nan, cls_pos_loss: nan, cls_neg_loss: nan, dir_loss_reduced: nan, cls_loss_reduced: nan, loc_loss_reduced: nan, loc_loss_elem: ['nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan', 'nan'], num_pos: 13.6200, num_neg: 63489.2200

xmyqsh commented 4 years ago

Reduce the learning rate?

chowkamlee81 commented 4 years ago

Where to find learning rate in code? Any links

chowkamlee81 commented 4 years ago

@xmyqsh Kindly help. @poodarchu

chowkamlee81 commented 4 years ago

made div_factor=100,10000 from 10.0. Still nan Values . Plesae hekp @xmyqsh , @poodarchu

a157801 commented 4 years ago

Please install nuscene-devkit following the installation and create data again. Official nuscenes-devkit will lead to this phenomenon.

zxduan90 commented 4 years ago

So is this a bug in the offical nuscenes-devkit or you just revise some code for your dataset? @a157801

muzi2045 commented 4 years ago

refer this https://github.com/poodarchu/Det3D/issues/19

xmyqsh commented 4 years ago

@a157801 It is better to give a benchmark of group class balancing on KITTI or a subset of nuScene. Most of us are CPU urging.

a157801 commented 4 years ago

So is this a bug in the offical nuscenes-devkit or you just revise some code for your dataset? @a157801

It is not a bug, we add velocity as an element of box when applying get_boxes function.

a157801 commented 4 years ago

refer this #19 Please install the nuscenes-devkit following install.md

a157801 commented 4 years ago

urging

Group class balancing aims to overcome the imbalance among a large number of categories, so it does not work on KITTI.