Closed ryanmaxwell96 closed 3 years ago
Also, the pretrained models I think should go into experiments, not src. Or at least that is what is working initially for me. It could not find the pretrained models in a models folder in src
Hi, Thanks for your questions! sorry about the confusion, but mot17_tracking.sh is only to provide example how to run the training and testing for DEFT, and not to run it directly. We have updated the requirements.txt with "cython_bbox". For the pretrained models, you can put them in src/models and follow mot17_tracking.sh and also you can put then in any other directory and just change the path when running the testing code. Please let me know if you have any additional questions. Thanks,
Ok that makes sense. I can use get_mot_17.sh correct?
Also, is there a reason why train.py will not run on GPU? Do you know what would be causing this and how to force everything onto GPU?
yes you can use use get_mot_17.sh. For the GPU using --gpus 0 when runing the code should run it on the GPU if a GPU is available.
In my conda environment, nvidia-smi returns CUDA 10.1. And nvcc --version returns CUDA 10.1. When I run sh mot17_tracking.sh, I get:
Running tracking Using tracking threshold for out threshold! 0.3 Fix size testing. training chunk_sizes: [4] Namespace(AFE=True, K=100, add_05=False, amodel_offset_weight=1, arch='dla_34', aug_rot=0, backbone='dla34', batch_size=4, chunk_sizes=[4], custom_dataset_ann_path='', custom_dataset_img_path='', data_dir='/home/maxwelr/DEFT/src/lib/../../data', dataset='mot', dataset_version='17trainval', debug=0, debug_dir='/home/maxwelr/DEFT/src/lib/../../exp/tracking/mot17_train/debug', debugger_theme='white', dense_reg=1, dep_weight=1, depth_scale=1, dim_weight=1, dla_node='dcn', down_ratio=4, efficient_level=0, eval_val=False, exp_dir='/home/maxwelr/DEFT/src/lib/../../exp/tracking', exp_id='mot17_train', fix_res=True, fix_short=-1, flip=0.5, flip_test=False, fp_disturb=0.1, gpus=[0], gpus_str='0', head_conv={'hm': [256], 'reg': [256], 'wh': [256], 'tracking': [256], 'ltrb_amodal': [256]}, head_kernel=3, heads={'hm': 1, 'reg': 2, 'wh': 2, 'tracking': 2, 'ltrb_amodal': 4}, hm_disturb=0.05, hm_hp_weight=1, hm_weight=1, hp_weight=1, hungarian=False, ignore_loaded_cats=[], input_h=544, input_res=960, input_w=960, keep_res=False, kitti_split='3dop', load_model='models/model_mot.pth', load_model_traj='', load_results='', lost_disturb=0.4, lr=0.000125, lr_step=[60], lstm=False, ltrb=False, ltrb_amodal=True, ltrb_amodal_weight=0.1, ltrb_weight=0.1, map_argoverse_id=False, master_batch_size=4, matching_weight=1, max_age=-1, max_frame_dist=1, max_frame_dist_AFE=10, max_object=100, model_output_list=False, msra_outchannel=256, neck='dlaup', new_thresh=0.3, nms=False, no_color_aug=False, no_pre_img=False, non_block_test=False, not_cuda_benchmark=False, not_idaup=False, not_max_crop=False, not_prefetch_test=False, not_rand_crop=False, not_set_cuda_env=False, not_show_bbox=False, not_show_number=False, not_show_txt=False, num_classes=1, num_epochs=80, num_head_conv=1, num_iters=-1, num_layers=101, num_stacks=1, num_workers=8, nuscenes_att=False, nuscenes_att_weight=1, off_weight=1, only_show_dots=False, optim='adam', out_thresh=0.3, output_h=136, output_res=240, output_w=240, pad=31, pre_hm=True, pre_img=True, pre_thresh=0.3, print_iter=0, prior_bias=-4.6, public_det=False, qualitative=False, reg_loss='l1', reset_hm=False, resize_video=False, resume=False, reuse_hm=False, root_dir='/home/maxwelr/DEFT/src/lib/../..', rot_weight=1, rotate=0, same_aug_pre=True, save_all=False, save_dir='/home/maxwelr/DEFT/src/lib/../../exp/tracking/mot17_train', save_framerate=30, save_img_suffix='', save_imgs=[], save_point=[90], save_results=False, save_video=False, scale=0, seed=317, shift=0, show_trace=False, show_track_color=False, skip_first=-1, tango_color=False, task='tracking', test=False, test_dataset='mot', test_focal_length=-1, test_scales=[1.0], track_buffer=30, track_thresh=0.3, tracking=True, tracking_weight=1, trainval=False, transpose_video=False, use_kpt_center=False, use_loaded_results=False, val_intervals=10000, velocity=False, velocity_weight=1, video_h=512, video_w=512, vis_gt_bev='', vis_thresh=0.3, weights={'hm': 1, 'reg': 1, 'wh': 0.1, 'tracking': 1, 'ltrb_amodal': 0.1}, wh_weight=0.1, zero_pre_hm=False, zero_tracking=False) Creating model... Warning: No ImageNet pretrain!! loaded models/model_mot.pth, epoch 70 Setting up train data... Using MOT 17 17trainval ann_file train.json ==> initializing train data from /home/maxwelr/DEFT/src/lib/../../data/mot17/annotations/train.json, images from /home/maxwelr/DEFT/src/lib/../../data/mot17/train ... loading annotations into memory... Done (t=0.34s) creating index... index created! Creating video index! Loaded MOT 17trainval train 5316 samples Starting training... yesyes tracking/mot17_train
And it remains there doing nothing. I have checked system monitor and some CPU cores are at 100%. When I run watch -n l nvidia-smi, I get the following output:
Python is still on CPU!
What do G and C types mean in nvidia-smi? https://stackoverflow.com/questions/54750627/what-do-g-and-c-types-mean-in-nvidia-smi
I get "No such file or directory" errors for test.py, train_predictions.py, and train.py when I run ./mot17_tracking.sh.
I see that these files are in DEFT/src so I change the file mot17_tracking.sh to direct it to find these files. Then I get "No module named 'cyuthon_bbox'".
Also, btw, I am missing gt_train_half.txt, gt_val_half.txt, det_train_half.txt, and det_val_half.txt as well as train_half.json and test_half.json.
Is this a problem?