[Testing] Computing the Accuracy Performance with Pre-Trained Standard COCO Dataset with 50 Batch and 300 Epochs on Server, Ubuntu Machine, Raspberry Pi 4, and Jetsan Nano

jaskiratsingh2000 commented 3 years ago

Server

Accuracy Performance (mAP) computed on Server:

mAP value = 0.564

val: data=./data/coco.yaml, weights=['yolov5s.pt'], batch_size=32, imgsz=640, conf_thres=0.001, iou_thres=0.6, task=val, device=, single_cls=False, augment=False, verbose=False, save_txt=False, save_hybrid=False, save_conf=False, save_json=True, project=runs/val, name=exp, exist_ok=False, half=False
/home/arani/anaconda3/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Unexpected error from cudaGetDev
iceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibil
ity was attempted on non supported HW (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
YOLOv5 🚀 v5.0-290-g62409ee torch 1.8.1+cu102 CPU
Fusing layers... 
Model Summary: 224 layers, 7266973 parameters, 0 gradients, 17.0 GFLOPs
val: Scanning '../datasets/coco/val2017.cache' images and labels... 4952 found, 48 missing, 0 empty, 0 corrupted: 100%|█| 5000/5000 [00:00<?, ?
             Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|████████████████| 157/157 [07:29<00:00,  2.87s/it]
                 all       5000      36335      0.663      0.521      0.564      0.363
Speed: 0.4ms pre-process, 76.0ms inference, 8.4ms NMS per image at shape (32, 3, 640, 640)

Evaluating pycocotools mAP... saving runs/val/exp2/best_predictions.json...
loading annotations into memory...
Done (t=0.44s)
creating index...
index created!
Loading and preparing results...
DONE (t=6.85s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=98.08s).
Accumulating evaluation results...
DONE (t=17.18s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.376
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.572
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.405
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.222
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.428
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.476
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.309
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.517
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.568
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.377
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.629
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.708

jaskiratsingh2000 commented 3 years ago

Ubuntu Machine

Accuracy Performance (mAP) computed on Ubuntu Machine:

mAP value = 0.564

               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|██████████████| 157/157 [1:49:30<00:00, 41.85s/it]
                 all       5000      36335      0.668      0.518      0.564      0.363
Speed: 4.0ms pre-process, 1273.3ms inference, 21.3ms NMS per image at shape (32, 3, 640, 640)

jaskiratsingh2000 commented 3 years ago

Raspberry Pi 4

Accuracy Performance (mAP) computed on Raspberry Pi 4:

mAP value = 0.000127

val: Scanning '../datasets/coco/val2017' images and labels...4952 found, 48
val: New cache created: ../datasets/coco/val2017.cache
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.               Class     Images     Labels          P          R     mAP@.               Class     Images     Labels          P          R     mAP@.               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5WARNING: NMS time limit 10.0s exceeded
               Class     Images     Labels          P          R     mAP@.5
                 all       5000      36335    0.00163    0.00058   0.000127   6.94e-05
Speed: 6.2ms pre-process, 2178.0ms inference, 248.3ms NMS per image at shape (32, 3, 640, 640)

Evaluating pycocotools mAP... saving runs/val/exp6/best_predictions.json...
loading annotations into memory...
Done (t=1.73s)
creating index...
index created!
Loading and preparing results...
DONE (t=20.32s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=125.62s).
Accumulating evaluation results...
DONE (t=55.08s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.001
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.001
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

jaskiratsingh2000 commented 3 years ago

val: data=./data/coco.yaml, weights=['runs/train/exp/weights/best.pt'], batch_size=32, imgsz=640, conf_thres=0.001, iou_thres=0.6, task=val, device=, single_cls=False, augment=False, verbose=False, save_txt=False, save_hybrid=False, save_conf=False, save_json=True, project=runs/val, name=exp, exist_ok=False, half=False
YOLOv5 🚀 v5.0-290-g62409ee torch 1.7.0a0+e85d494 CPU

Fusing layers... 
Model Summary: 224 layers, 7266973 parameters, 0 gradients, 17.0 GFLOPs
val: Scanning '../datasets/coco/val2017.cache' images and labels... 4952 fo
               Class     Images     Labels          P          R     mAP@.5
               Class     Images     Labels          P          R     mAP@.5
                 all       5000      36335      0.053    0.00138   0.000123   4.76e-05
Speed: 7.2ms pre-process, 1578.8ms inference, 502.9ms NMS per image at shape (32, 3, 640, 640)

Evaluating pycocotools mAP... saving runs/val/exp8/best_predictions.json...
loading annotations into memory...
Done (t=1.74s)
creating index...
index created!
Loading and preparing results...
DONE (t=32.39s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=330.25s).
Accumulating evaluation results...
DONE (t=90.23s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.001
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.001
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001
Results saved to runs/val/exp8

jaskiratsingh2000 commented 3 years ago

python3 val.py --data coco128.yaml --weights runs/train/exp8/weights/best.pt val: data=./data/coco128.yaml, weights=['runs/train/exp8/weights/best.pt'], batch_size=32, imgsz=640, conf_thres=0.001, iou_thres=0.6, task=val, device=, single_cls=False, augment=False, verbose=False, save_txt=False, save_hybrid=False, save_conf=False, save_json=False, project=runs/val, name=exp, exist_ok=False, half=False YOLOv5 🚀 v5.0-371-gf3e3f76 torch 1.7.0a0+e85d494 CPU

Fusing layers... Model Summary: 224 layers, 7266973 parameters, 0 gradients, 17.0 GFLOPs val: Scanning '../datasets/coco128/labels/train2017.cache' images and label Class Images Labels P R mAP@.5 all 128 929 6.97e-05 0.00586 1.83e-05 3.01e-06 Speed: 10.3ms pre-process, 1925.2ms inference, 57.0ms NMS per image at shape (32, 3, 640, 640) Results saved to runs/val/exp

jaskiratsingh2000 / Research-Scalable-Vehicle-Detection-on-Edge-Devices