Open sparshgarg23 opened 2 years ago
Wanted to check if this is an issue with mmdetection or with mmtrack?
Hi, this would be caused by the update of the mmdet. You can try to add the cfg.device in the train.py as in https://github.com/open-mmlab/mmdetection/blob/master/tools/train.py#L196
Please let me know if this helps.
Was able to proceed with training after making the suggested changes,but it seems that the training process is currently stuck at this stage
Loading and preparing results...
DONE (t=1.68s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
Any reason why this is happening?
Was able to resolve the previous issue,it seems that evaluation of annotation type bbox takes 21 minutes per epoch. I am currently training the reid model on MOT20,and it seems that for each epoch it's processing about 76139 batches. Since I am training on colab,is there a way to speed up the process.Increasing the batch size beyond 8 results in cuda out of memory error
It seems that the speed is limited by the GPU memory in your case.
well that would make sense.I am working on colab so I just have access to one GPu @dyhBUPT could you tell me how to go about fixing #542 . I was able to obtain the demo results by using the pretrained detection and reid models for traktor. But when I used my own detection model I ended up getting key error detections when I tried to do an inference
Thanks for your error report and we appreciate it a lot.
Checklist
I ran into the following error
TorchVision: 0.12.0+cu113 OpenCV: 4.1.2 MMCV: 1.5.0 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 11.1 MMTracking: 0.13.0+88f92dd
2022-05-02 11:26:04,231 - mmtrack - INFO - Distributed training: False 2022-05-02 11:26:04,890 - mmtrack - INFO - Config: model = dict( type='FasterRCNN', backbone=dict( type='ResNet', depth=50, num_stages=4, out_indices=(0, 1, 2, 3), frozen_stages=1, norm_cfg=dict(type='BN', requires_grad=True), norm_eval=True, style='pytorch', init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50')), neck=dict( type='FPN', in_channels=[256, 512, 1024, 2048], out_channels=256, num_outs=5), rpn_head=dict( type='RPNHead', in_channels=256, feat_channels=256, anchor_generator=dict( type='AnchorGenerator', scales=[8], ratios=[0.5, 1.0, 2.0], strides=[4, 8, 16, 32, 64]), bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[1.0, 1.0, 1.0, 1.0], clip_border=True), loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0), loss_bbox=dict( type='SmoothL1Loss', beta=0.1111111111111111, loss_weight=1.0)), roi_head=dict( type='StandardRoIHead', bbox_roi_extractor=dict( type='SingleRoIExtractor', roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0), out_channels=256, featmap_strides=[4, 8, 16, 32]), bbox_head=dict( type='Shared2FCBBoxHead', in_channels=256, fc_out_channels=1024, roi_feat_size=7, num_classes=1, bbox_coder=dict( type='DeltaXYWHBBoxCoder', target_means=[0.0, 0.0, 0.0, 0.0], target_stds=[0.1, 0.1, 0.2, 0.2], clip_border=True), reg_class_agnostic=False, loss_cls=dict( type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0), loss_bbox=dict(type='SmoothL1Loss', loss_weight=1.0))), train_cfg=dict( rpn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.7, neg_iou_thr=0.3, min_pos_iou=0.3, match_low_quality=True, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=256, pos_fraction=0.5, neg_pos_ub=-1, add_gt_as_proposals=False), allowed_border=-1, pos_weight=-1, debug=False), rpn_proposal=dict( nms_pre=2000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( assigner=dict( type='MaxIoUAssigner', pos_iou_thr=0.5, neg_iou_thr=0.5, min_pos_iou=0.5, match_low_quality=False, ignore_iof_thr=-1), sampler=dict( type='RandomSampler', num=512, pos_fraction=0.25, neg_pos_ub=-1, add_gt_as_proposals=True), pos_weight=-1, debug=False)), test_cfg=dict( rpn=dict( nms_pre=1000, max_per_img=1000, nms=dict(type='nms', iou_threshold=0.7), min_bbox_size=0), rcnn=dict( score_thr=0.05, nms=dict(type='nms', iou_threshold=0.5), max_per_img=100)), init_cfg=dict( type='Pretrained', checkpoint= 'http://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth' )) dataset_type = 'CocoDataset' img_norm_cfg = dict( mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True) train_pipeline = [ dict(type='LoadImageFromFile', to_float32=True), dict(type='LoadAnnotations', with_bbox=True), dict( type='Resize', img_scale=(1088, 1088), ratio_range=(0.8, 1.2), keep_ratio=True, bbox_clip_border=False), dict(type='PhotoMetricDistortion'), dict(type='RandomCrop', crop_size=(1088, 1088), bbox_clip_border=False), dict(type='RandomFlip', flip_ratio=0.5), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']) ] test_pipeline = [ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1088, 1088), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ] data_root = 'data/MOT20/' data = dict( samples_per_gpu=8, workers_per_gpu=8, train=dict( type='CocoDataset', ann_file='data/MOT20/annotations/half-train_cocoformat.json', img_prefix='data/MOT20/train', classes=('pedestrian', ), pipeline=[ dict(type='LoadImageFromFile', to_float32=True), dict(type='LoadAnnotations', with_bbox=True), dict( type='Resize', img_scale=(1088, 1088), ratio_range=(0.8, 1.2), keep_ratio=True, bbox_clip_border=False), dict(type='PhotoMetricDistortion'), dict( type='RandomCrop', crop_size=(1088, 1088), bbox_clip_border=False), dict(type='RandomFlip', flip_ratio=0.5), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='DefaultFormatBundle'), dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']) ]), val=dict( type='CocoDataset', ann_file='data/MOT20/annotations/half-val_cocoformat.json', img_prefix='data/MOT20/train', classes=('pedestrian', ), pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1088, 1088), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ]), test=dict( type='CocoDataset', ann_file='data/MOT20/annotations/half-val_cocoformat.json', img_prefix='data/MOT20/train', classes=('pedestrian', ), pipeline=[ dict(type='LoadImageFromFile'), dict( type='MultiScaleFlipAug', img_scale=(1088, 1088), flip=False, transforms=[ dict(type='Resize', keep_ratio=True), dict(type='RandomFlip'), dict( type='Normalize', mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True), dict(type='Pad', size_divisor=32), dict(type='ImageToTensor', keys=['img']), dict(type='Collect', keys=['img']) ]) ])) evaluation = dict(metric=['bbox']) optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001) optimizer_config = dict(grad_clip=None) checkpoint_config = dict(interval=1) log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')]) dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)] opencv_num_threads = 0 mp_start_method = 'fork' USE_MMDET = True lr_config = dict( policy='step', warmup='linear', warmup_iters=100, warmup_ratio=0.01, step=[6]) total_epochs = 8 work_dir = './work_dirs/faster-rcnn_r50_fpn_8e_mot20-half' gpu_ids = [0]
2022-05-02 11:26:04,933 - mmtrack - INFO - Set random seed to 849712315, deterministic: False 2022-05-02 11:26:05,370 - mmtrack - INFO - initialize FasterRCNN with init_cfg {'type': 'Pretrained', 'checkpoint': 'http://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth'} 2022-05-02 11:26:05,370 - mmcv - INFO - load model from: http://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth 2022-05-02 11:26:05,371 - mmcv - INFO - load checkpoint from http path: http://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth Downloading: "http://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_2x_coco/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth" to /root/.cache/torch/hub/checkpoints/faster_rcnn_r50_fpn_2x_coco_bbox_mAP-0.384_20200504_210434-a5d8aa15.pth 100% 160M/160M [00:16<00:00, 10.4MB/s] 2022-05-02 11:26:22,426 - mmcv - WARNING - The model and loaded state dict do not match exactly
size mismatch for roi_head.bbox_head.fc_cls.weight: copying a param with shape torch.Size([81, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]). size mismatch for roi_head.bbox_head.fc_cls.bias: copying a param with shape torch.Size([81]) from checkpoint, the shape in current model is torch.Size([2]). size mismatch for roi_head.bbox_head.fc_reg.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([4, 1024]). size mismatch for roi_head.bbox_head.fc_reg.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([4]). loading annotations into memory... Done (t=3.20s) creating index... index created! /usr/local/lib/python3.7/dist-packages/mmdet/utils/compat_config.py:30: UserWarning: config is now expected to have a
main()
File "tools/train.py", line 206, in main
meta=meta)
File "/usr/local/lib/python3.7/dist-packages/mmdet/apis/train.py", line 163, in train_detector
model = build_dp(model, cfg.device, device_ids=cfg.gpu_ids)
File "/usr/local/lib/python3.7/dist-packages/mmcv/utils/config.py", line 513, in getattr
return getattr(self._cfg_dict, name)
File "/usr/local/lib/python3.7/dist-packages/mmcv/utils/config.py", line 49, in getattr
raise ex
AttributeError: 'ConfigDict' object has no attribute 'device'
runner
section, please setrunner
in your config. 'please setrunner
in your config.', UserWarning) /usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked)) Traceback (most recent call last): File "tools/train.py", line 210, inExecute !python ./tools/convert_datasets/mot/mot2coco.py -i ./data/MOT20/ -o ./data/MOT20/annotations --split-train --convert-det
Convert mot2reid using !python ./tools/convert_datasets/mot/mot2reid.py -i ./data/MOT20/ -o ./data/MOT20/reid --val-split 0.2 --vis-threshold 0.3 3.Train using !python tools/train.py /content/mmtracking/configs/det/faster-rcnn_r50_fpn_8e_mot20-half.py
Did you make any modifications on the code or config? Did you understand what you have modified? only changed number of samples_per_gpu from 2 to 8 in datasets/mot_challenge_det.py
What dataset did you use and what task did you run? MOT20 multiobject tracking first train model for detection as mentioned in quick_run.md Environment Environment information
TorchVision: 0.12.0+cu113 OpenCV: 4.1.2 MMCV: 1.5.0 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 11.1 MMTracking: 0.13.0+88f92dd