search

yang-0201 / MAF-YOLO

Implementation of paper - Multi-Branch Auxiliary Fusion YOLO with Re-parameterization Heterogeneous Convolutional for accurate object detection.
GNU General Public License v3.0
46 stars 5 forks source link

mAP calculated by pycocotools collapsed #2

Open XYQDUT opened 1 week ago

XYQDUT commented 1 week ago
  1. I saw in the paper that this method improves small object detection. However, when I trained with the VisDrone dataset, I found that the mAP calculated by pycocotools collapsed. Here is the output from the last epoch of training.

# Evaluating speed.

Evaluating mAP by pycocotools. Saving MAF-YOLO/exp4/predictions.json... Results saved to MAF-YOLO/exp4 Epoch: 99 | mAP@0.5: 0.03880035055291523 | mAP@0.50:0.95: 0.017914316025343748 best mAP@0.50:0.95: 0.04084159832035066 | best epoch: 0

Training completed in 4.064 hours. loading annotations into memory... Done (t=0.13s) creating index... index created! Loading and preparing results... DONE (t=1.07s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=23.53s). Accumulating evaluation results... DONE (t=1.74s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.018 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.039 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.015 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.019 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.033 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.012 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.060 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.098 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.098 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.117 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.150 #

  1. However, using the weight file last_ckpt.pt saved from the final training session results in a better mAP. Here is the result.

# Class Images Labels P@.5iou R@.5iou F1@.5iou mAP@.5 mAP@.5:.95 all 548 38759 0.591 0.43 0.492 0.436 0.261

Evaluating speed. Average pre-process time: 0.24 ms Average inference time: 6.46 ms Average NMS time: 3.61 ms

Evaluating mAP by pycocotools. Saving runs/val/result/predictions.json... loading annotations into memory... Done (t=0.29s) creating index... index created! Loading and preparing results... DONE (t=1.13s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=19.56s). Accumulating evaluation results... DONE (t=1.64s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.018 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.039 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.015 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.019 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.033 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.012 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.059 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.098 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.098 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.116 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.150 #

yang-0201 commented 1 week ago

Thank you very much for your interest in this project，Is the training log in the middle working properly？ How accurate is it? I'd like you to provide me with the model you used and the commands you used to train it.

XYQDUT commented 1 week ago

Thank you very much for your interest in this project，Is the training log in the middle working properly？ How accurate is it? I'd like you to provide me with the model you used and the commands you used to train it.

Except for the results calculated by pycocotools, everything else seems normal. The training command python tools/train.py --data-path data/Visdrone.yaml --conf-file configs/pretrain/MAF-YOLO-m-pretrain.py

yang-0201 commented 4 days ago

Hello, I re-uploaded a new version of 1.1.0 and did not have a similar problem, you can re-download the code and try again!

yang-0201 commented 4 days ago

train cmd: python tools/train.py --conf configs/pretrain/MAF-YOLO-m-pretrain.py --data data/VisDrone.yaml --device 0 --batch 8 --epochs 100 get:

transform_weight: 1562/1568 from MAFYOLOm.pt transform_model_weight: 1562/1568 from MAFYOLOm.pt

Training start...

 Epoch  iou_loss  dfl_loss  cls_loss
  0/99    0.6247     0.375     1.288: 100%|██████████| 809/809 [04:17<00:00,  3.14it/s]

Inferencing model in train datasets.: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:17<00:00, 2.02it/s]

Evaluating speed.

Evaluating mAP by pycocotools. Saving MAF-YOLO\exp1\predictions.json... loading annotations into memory... Done (t=0.10s) creating index... index created! Loading and preparing results... DONE (t=0.83s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=19.35s). Accumulating evaluation results... DONE (t=1.16s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.126 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.209 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.127 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.059 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.211 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.357 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.068 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.215 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.296 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.177 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.474 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.656 Results saved to MAF-YOLO\exp1 Epoch: 0 | mAP@0.5: 0.20894816711035757 | mAP@0.50:0.95: 0.1255171991137001 best mAP@0.50:0.95: 0.1255171991137001 | best epoch: 0

 Epoch  iou_loss  dfl_loss  cls_loss
  1/99    0.5766    0.3498    0.9683: 100%|██████████| 809/809 [04:00<00:00,  3.37it/s]

Inferencing model in train datasets.: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:15<00:00, 2.23it/s]

Evaluating speed.

Evaluating mAP by pycocotools. Saving MAF-YOLO\exp1\predictions.json... loading annotations into memory... Done (t=0.09s) creating index... index created! Loading and preparing results... DONE (t=0.78s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=19.79s). Accumulating evaluation results... DONE (t=1.08s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.159 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.264 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.161 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.073 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.267 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.461 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.082 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.234 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.308 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.186 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.494 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.713 Results saved to MAF-YOLO\exp1 Epoch: 1 | mAP@0.5: 0.2639859069451592 | mAP@0.50:0.95: 0.1586531816945784 best mAP@0.50:0.95: 0.1586531816945784 | best epoch: 1

 Epoch  iou_loss  dfl_loss  cls_loss
  2/99    0.5607    0.3416    0.9162: 100%|██████████| 809/809 [03:55<00:00,  3.43it/s]

Inferencing model in train datasets.: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:15<00:00, 2.19it/s]

Evaluating speed.

Evaluating mAP by pycocotools. Saving MAF-YOLO\exp1\predictions.json... loading annotations into memory... Done (t=0.10s) creating index... index created! Loading and preparing results... DONE (t=0.88s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=19.76s). Accumulating evaluation results... DONE (t=1.13s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.182 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.303 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.186 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.088 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.299 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.468 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.089 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.259 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.340 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.216 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.526 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.736 Results saved to MAF-YOLO\exp1 Epoch: 2 | mAP@0.5: 0.3027632490114197 | mAP@0.50:0.95: 0.18194999742836987 best mAP@0.50:0.95: 0.18194999742836987 | best epoch: 2

XYQDUT commented 4 days ago

Thank you very much. I will download and re-run the program.

发件人: zhiqiang yang @.> 发送时间: 2024年9月11日 16:30 收件人: yang-0201/MAF-YOLO @.> 抄送: xyq @.>; Author @.> 主题: Re: [yang-0201/MAF-YOLO] mAP calculated by pycocotools collapsed (Issue #2)

Hello, I re-uploaded a new version of 1.1.0 and did not have a similar problem, you can re-download the code and try again!

― Reply to this email directly, view it on GitHubhttps://github.com/yang-0201/MAF-YOLO/issues/2#issuecomment-2342994640, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BCYE6RRO46EFWSR7343MO63ZV75RJAVCNFSM6AAAAABNP24NL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBSHE4TINRUGA. You are receiving this because you authored the thread.Message ID: @.***>