xingyizhou
commented
4 years ago Thanks for training our model for such a long time. The reason behind the size difference is my provided model does not include the optimization state duct (which is not used in testing), and the trained model contains it for continuing training.
There might be a few concerns about training nuScenes with a single 12GB GPU. First, when bachsize is changed, the learning rate needs to be changed as well according to the linear learning rate rule. Also, using a small batchsize makes the batchnorm layer less effective or sometimes harmful. I would say it will be very hard to reproduce our results with one single 12GB GPU.
qingnanli
yustaub
Hi, I appreciate your work, but got an error. I changed the batch_size to 12 since I have only one GPU (pytorch 1.2.0):
python main.py tracking,ddd --exp_id nuScenes_3Dtracking --dataset nuscenes --pre_hm --load_model ../models/nuScenes_3Ddetection_e140.pth --shift 0.01 --scale 0.05 --lost_disturb 0.4 --fp_disturb 0.1 --hm_disturb 0.05 --batch_size 12 --gpus 0 --lr 2.5e-4 --save_point 60
tracking,ddd/nuScenes_3Dtracking |############################### | train: [70][14057/14065]|Tot: 5:58:18 |ETA: 0:00:13 |tot 23.5435 |hm 2.5172 |wh 5.9240 |reg 0.2477tracking,ddd/nuScenes_3Dtracking |############################### | train: [70][14058/14065]|Tot: 5:58:20 |ETA: 0:00:11 |tot 23.5445 |hm 2.5172 |wh 5.9243 |reg 0.2477tracking,ddd/nuScenes_3Dtracking |############################### | train: [70][14059/14065]|Tot: 5:58:21 |ETA: 0:00:10 |tot 23.5444 |hm 2.5172 |wh 5.9242 |reg 0.2477tracking,ddd/nuScenes_3Dtracking |############################### | train: [70][14060/14065]|Tot: 5:58:23 |ETA: 0:00:08 |tot 23.5445 |hm 2.5172 |wh 5.9242 |reg 0.2477tracking,ddd/nuScenes_3Dtracking |############################### | train: [70][14061/14065]|Tot: 5:58:24 |ETA: 0:00:07 |tot 23.5445 |hm 2.5172 |wh 5.9242 |reg 0.2477tracking,ddd/nuScenes_3Dtracking |############################### | train: [70][14062/14065]|Tot: 5:58:26 |ETA: 0:00:05 |tot 23.5441 |hm 2.5172 |wh 5.9241 |reg 0.2477tracking,ddd/nuScenes_3Dtracking |############################### | train: [70][14063/14065]|Tot: 5:58:27 |ETA: 0:00:04 |tot 23.5441 |hm 2.5172 |wh 5.9245 |reg 0.2477tracking,ddd/nuScenes_3Dtracking |################################| train: [70][14064/14065]|Tot: 5:58:29 |ETA: 0:00:02 |tot 23.5440 |hm 2.5173 |wh 5.9244 |reg 0.2477 |dep 8.6710 |dim 0.7938 |rot 2.1158 |amodel_offset 3.0259 |tracking 5.5801 |Data 0.001s(0.001s) |Net 1.529s
About 2 weeks training, I got model_last.pth (244.8 MB) and model_60.pth (244.8 MB) in dir exp/tracking.ddd/nuScenes_3Dtracking, however, your provided nuScenes_3Dtracking.pth is only 81.8 MB. I use nuScenes_3Dtracking.pth and get the expected AMOTA is 6.8, but if I use my own trained model_last.pth or model_60.pth or command resume:
python test.py tracking,ddd --exp_id nuScenes_3Dtracking --load_model ../models/model_60.pth --dataset nuscenes --track_thresh 0.1 --pre_hm
OR
python test.py tracking,ddd --exp_id nuScenes_3Dtracking --dataset nuscenes --pre_hm --track_thresh 0.1 --resume
I got the following errors:
--track_thresh 0.1 --resume /home/qingqing/.conda/envs/CenterTrack/lib/python3.6/site-packages/sklearn/utils/linearassignment.py:22: FutureWarning: The linearassignment module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead. FutureWarning) Running tracking Using tracking threshold for out threshold! 0.1 Fix size testing. training chunk_sizes: [32] input h w: 448 800 heads {'hm': 10, 'reg': 2, 'wh': 2, 'tracking': 2, 'dep': 1, 'rot': 8, 'dim': 3, 'amodel_offset': 2} weights {'hm': 1, 'reg': 1, 'wh': 0.1, 'tracking': 1, 'dep': 1, 'rot': 1, 'dim': 1, 'amodel_offset': 1} head conv {'hm': [256], 'reg': [256], 'wh': [256], 'tracking': [256], 'dep': [256], 'rot': [256], 'dim': [256], 'amodel_offset': [256]} Namespace(K=100, add_05=False, amodel_offset_weight=1, arch='dla_34', aug_rot=0, backbone='dla34', batch_size=32, chunk_sizes=[32], custom_dataset_ann_path='', custom_dataset_img_path='', data_dir='/home/qingqing/Downloads/qingqing_disk/p4600_disk/CenterTrack/src/lib/../../data', dataset='nuscenes', dataset_version='', debug=0, debug_dir='/home/qingqing/Downloads/qingqing_disk/p4600_disk/CenterTrack/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking/debug', debugger_theme='white', demo='', dense_reg=1, dep_weight=1, depth_scale=1, dim_weight=1, dla_node='dcn', down_ratio=4, efficient_level=0, eval_val=False, exp_dir='/home/qingqing/Downloads/qingqing_disk/p4600_disk/CenterTrack/src/lib/../../exp/tracking,ddd', exp_id='nuScenes_3Dtracking', fix_res=True, fix_short=-1, flip=0.5, flip_test=False, fp_disturb=0, gpus=[0], gpus_str='0', head_conv={'hm': [256], 'reg': [256], 'wh': [256], 'tracking': [256], 'dep': [256], 'rot': [256], 'dim': [256], 'amodel_offset': [256]}, head_kernel=3, heads={'hm': 10, 'reg': 2, 'wh': 2, 'tracking': 2, 'dep': 1, 'rot': 8, 'dim': 3, 'amodel_offset': 2}, hm_disturb=0, hm_hp_weight=1, hm_weight=1, hp_weight=1, hungarian=False, ignore_loaded_cats=[], input_h=448, input_res=800, input_w=800, keep_res=False, kitti_split='3dop', load_model='/home/qingqing/Downloads/qingqing_disk/p4600_disk/CenterTrack/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking/model_last.pth', load_results='', lost_disturb=0, lr=0.000125, lr_step=[60], ltrb=False, ltrb_amodal=False, ltrb_amodal_weight=0.1, ltrb_weight=0.1, map_argoverse_id=False, master_batch_size=32, max_age=-1, max_frame_dist=3, model_output_list=False, msra_outchannel=256, neck='dlaup', new_thresh=0.3, nms=False, no_color_aug=False, no_pause=False, no_pre_img=False, non_block_test=False, not_cuda_benchmark=False, not_idaup=False, not_max_crop=False, not_prefetch_test=False, not_rand_crop=False, not_set_cuda_env=False, not_show_bbox=False, not_show_number=False, num_classes=10, num_epochs=70, num_head_conv=1, num_iters=-1, num_layers=101, num_stacks=1, num_workers=4, nuscenes_att=False, nuscenes_att_weight=1, off_weight=1, optim='adam', out_thresh=0.1, output_h=112, output_res=200, output_w=200, pad=31, pre_hm=True, pre_img=True, pre_thresh=0.1, print_iter=0, prior_bias=-4.6, public_det=False, qualitative=False, reg_loss='l1', reset_hm=False, resize_video=False, resume=True, reuse_hm=False, root_dir='/home/qingqing/Downloads/qingqing_disk/p4600_disk/CenterTrack/src/lib/../..', rot_weight=1, rotate=0, same_aug_pre=False, save_all=False, save_dir='/home/qingqing/Downloads/qingqing_disk/p4600_disk/CenterTrack/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking', save_framerate=30, save_img_suffix='', save_imgs=[], save_point=[90], save_results=False, save_video=False, scale=0, seed=317, shift=0, show_track_color=True, skip_first=-1, tango_color=False, task='tracking,ddd', test=False, test_dataset='nuscenes', test_focal_length=-1, test_scales=[1.0], track_thresh=0.1, tracking=True, tracking_weight=1, trainval=False, transpose_video=False, use_kpt_center=False, use_loaded_results=False, val_intervals=10000, velocity=False, velocity_weight=1, video_h=512, video_w=512, vis_gt_bev='', vis_thresh=0.3, weights={'hm': 1, 'reg': 1, 'wh': 0.1, 'tracking': 1, 'dep': 1, 'rot': 1, 'dim': 1, 'amodel_offset': 1}, wh_weight=0.1, zero_pre_hm=False, zero_tracking=False) Dataset version ==> initializing val data from /home/qingqing/Downloads/qingqing_disk/p4600_disk/CenterTrack/src/lib/../../data/nuscenes/annotations/val.json, images from /home/qingqing/Downloads/qingqing_disk/p4600_disk/CenterTrack/src/lib/../../data/nuscenes/v1.0-trainval ... loading annotations into memory... Done (t=4.80s) creating index... index created! Creating video index! Loaded val 36114 samples Creating model... Using node type: (<class 'model.networks.dla.DeformConv'>, <class 'model.networks.dla.DeformConv'>) Warning: No ImageNet pretrain!! loaded /home/qingqing/Downloads/qingqing_disk/p4600_disk/CenterTrack/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking/model_last.pth, epoch 70 nuScenes_3Dtracking No pre_dets for 1 . Use empty initialization. Start tracking video 1 Initialize tracking!
......
......
......
85%|██████████████████████████████████████████████████████████████████████████████████████████ 86%|██████████████████████████████████████████████████████████████████████████████████████████ 88%|██████████████████████████████████████████████████████████████████████████████████████████ 89%|██████████████████████████████████████████████████████████████████████████████████████████ 91%|██████████████████████████████████████████████████████████████████████████████████████████ 96%|██████████████████████████████████████████████████████████████████████████████████████████ 99%|██████████████████████████████████████████████████████████████████████████████████████████100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6019/6019 [00:09<00:00, 639.51it/s] Loaded ground truth annotations for 6019 samples. Filtering tracks Traceback (most recent call last): File "tools/nuscenes-devkit/python-sdk-alpha02/nuscenes/eval/tracking/evaluate.py", line 271, in
render_classes=renderclasses)
File "tools/nuscenes-devkit/python-sdk-alpha02/nuscenes/eval/tracking/evaluate.py", line 97, in init
pred_boxes = filter_eval_boxes(nusc, pred_boxes, self.cfg.class_range, verbose=verbose)
File "/home/qingqing/.conda/envs/CenterTrack/lib/python3.6/site-packages/nuscenes/eval/common/loaders.py", line 220, in filter_eval_boxes
box = eval_boxes.boxes[first_key][0]
IndexError: list index out of range
Can anyone help?