search

BICLab / SpikeYOLO

Offical implementation of "Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detection" (ECCV2024 Best Paper Candidate / Oral)
https://arxiv.org/abs/2407.20708
GNU Affero General Public License v3.0
108 stars 6 forks source link

COCO evaluation results with pretrained weights #21

Closed HeinZingerRyo closed 1 day ago

HeinZingerRyo commented 1 day ago

Hi, I've downloaded your pretrained model weight 'best.pt' and ran test.py on COCO2017 validation set, but the performance is somehow different from the reported value in your paper: 微信图片_20241112163323 in which mAP with iou=0.50 is only 62.8%

To reproduce: Git clone this repo and set up the environment; downloaded COCO 2017 validation set and extracted; generated val2017.txt with all 5000 validation images; only modified some code for file paths e.g. model = YOLO('/path/to/weight/.pt') and run test.py

Here are some logs with regard to this result:

...
model.model.26.cv3.2.0.lif.qtrick
model.model.26.cv3.2.1.lif.qtrick
model.model.26.cv3.2.2.lif.qtrick
model.model.26.dfl.lif.qtrick
Ultralytics YOLOv8.0.197 🚀 Python-3.8.10 torch-2.0.0+cu118 CUDA:2 (NVIDIA GeForce RTX 4090 D, 24210MiB)
/root/miniconda3/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
snn_YOLOv8s summary: 621 layers, 23153024 parameters, 0 gradients, 136.9 GFLOPs
val: Scanning /root/xxx/dataset/coco/labels/val2017... 0 images, 5000 backgrounds, 0 corrupt: 100%|██████████| 5000/5000 [00:02<00:00, 2337.44it/s]
val: WARNING ⚠️ No labels found in /root/xxx/dataset/coco/labels/val2017.cache. See https://docs.ultralytics.com/datasets/detect for dataset formatting guidance.
val: WARNING ⚠️ Cache directory /root/xxx/dataset/coco/labels is not writeable, cache not saved.
WARNING ⚠️ No labels found in /root/xx/dataset/coco/labels/val2017.cache, training may not work correctly. See https://docs.ultralytics.com/datasets/detect for dataset formatting guidance.
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95): 100%|██████████| 250/250 [00:44<00:00,  5.62it/s]
                   all       5000          0          0          0          0          0
WARNING ⚠️ no labels found in detect set, can not compute metrics without labels
Speed: 0.1ms preprocess, 7.4ms inference, 0.0ms loss, 0.4ms postprocess per image
Saving /root/xxx/SpikeYOLO/runs/detect/val6/predictions.json...

Evaluating pycocotools mAP using /root/xxx/SpikeYOLO/runs/detect/val6/predictions.json and /root/xxx/dataset/coco/annotations/instances_val2017.json...
requirements: Ultralytics requirement ['pycocotools>=2.0.6'] not found, attempting AutoUpdate...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: http://mirrors.aliyun.com/pypi/simple
Collecting pycocotools>=2.0.6
  Downloading http://mirrors.aliyun.com/pypi/packages/6c/11/6cb76ebc71388ac17691bc3da76276d1642af30bf9097de9bb5f64c92cfa/pycocotools-2.0.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (439 kB)
Requirement already satisfied: matplotlib>=2.1.0 in /root/miniconda3/lib/python3.8/site-packages (from pycocotools>=2.0.6) (3.7.1)
Requirement already satisfied: numpy in /root/miniconda3/lib/python3.8/site-packages (from pycocotools>=2.0.6) (1.24.2)
Requirement already satisfied: contourpy>=1.0.1 in /root/miniconda3/lib/python3.8/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.6) (1.0.7)
Requirement already satisfied: packaging>=20.0 in /root/miniconda3/lib/python3.8/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.6) (23.0)
Requirement already satisfied: cycler>=0.10 in /root/miniconda3/lib/python3.8/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.6) (0.11.0)
Requirement already satisfied: python-dateutil>=2.7 in /root/miniconda3/lib/python3.8/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.6) (2.8.2)
Requirement already satisfied: pyparsing>=2.3.1 in /root/miniconda3/lib/python3.8/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.6) (3.0.9)
Requirement already satisfied: kiwisolver>=1.0.1 in /root/miniconda3/lib/python3.8/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.6) (1.4.4)
Requirement already satisfied: fonttools>=4.22.0 in /root/miniconda3/lib/python3.8/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.6) (4.39.0)
Requirement already satisfied: pillow>=6.2.0 in /root/miniconda3/lib/python3.8/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.6) (9.4.0)
Requirement already satisfied: importlib-resources>=3.2.0 in /root/miniconda3/lib/python3.8/site-packages (from matplotlib>=2.1.0->pycocotools>=2.0.6) (5.12.0)
Requirement already satisfied: zipp>=3.1.0 in /root/miniconda3/lib/python3.8/site-packages (from importlib-resources>=3.2.0->matplotlib>=2.1.0->pycocotools>=2.0.6) (3.15.0)
Requirement already satisfied: six>=1.5 in /root/miniconda3/lib/python3.8/site-packages (from python-dateutil>=2.7->matplotlib>=2.1.0->pycocotools>=2.0.6) (1.16.0)
Installing collected packages: pycocotools
Successfully installed pycocotools-2.0.7

requirements: AutoUpdate success ✅ 2.5s, installed 1 package: ['pycocotools>=2.0.6']
requirements: ⚠️ Restart runtime or rerun command for updates to take effect

loading annotations into memory...
Done (t=0.36s)
creating index...
index created!
Loading and preparing results...
DONE (t=1.37s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=35.11s).
Accumulating evaluation results...
DONE (t=9.45s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.456
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.628
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.500
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.257
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.506
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.616
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.361
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.593
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.646
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.445
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.712
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.809
Results saved to /root/xxx/SpikeYOLO/runs/detect/val6
fire: {'model.model.2.Conv.lif1.qtrick': tensor(2.3987, device='cuda:0'), 'model.model.2.Conv.lif2.qtrick': tensor(2.4457, device='cuda:0'), 'model.model.2.Conv.lif3.qtrick': tensor(2.0405, device='cuda:0'), 'model.model.2.conv1.lif.qtrick': tensor(2.9850, device='cuda:0'), 'model.model.2.conv2.lif.qtrick': tensor(0.5083, device='cuda:0'), 'model.model.3.encode_lif.qtrick': tensor(2.0617, device='cuda:0'), 'model.model.4.0.Conv.lif1.qtrick': tensor(1.5556, device='cuda:0'), 'model.model.4.0.Conv.lif2.qtrick': tensor(1.5254, device='cuda:0'), 'model.model.4.0.Conv.lif3.qtrick': tensor(0.8446, device='cuda:0'), 'model.model.4.0.conv1.lif.qtrick': tensor(1.7873, device='cuda:0'), 'model.model.4.0.conv2.lif.qtrick': tensor(0.2025, device='cuda:0'), 'model.model.4.1.Conv.lif1.qtrick': tensor(1.5432, device='cuda:0'), 'model.model.4.1.Conv.lif2.qtrick': tensor(0.8574, device='cuda:0'), 'model.model.4.1.Conv.lif3.qtrick': tensor(0.5343, device='cuda:0'), 'model.model.4.1.conv1.lif.qtrick': tensor(1.3349, device='cuda:0'), 'model.model.4.1.conv2.lif.qtrick': tensor(0.0984, device='cuda:0'), 'model.model.5.encode_lif.qtrick': tensor(1.6918, device='cuda:0'), 'model.model.6.0.Conv.lif1.qtrick': tensor(0.6383, device='cuda:0'), 'model.model.6.0.Conv.lif2.qtrick': tensor(0.7244, device='cuda:0'), 'model.model.6.0.Conv.lif3.qtrick': tensor(0.4541, device='cuda:0'), 'model.model.6.0.lif1.qtrick': tensor(0.6916, device='cuda:0'), 'model.model.6.0.lif2.qtrick': tensor(0.2237, device='cuda:0'), 'model.model.6.1.Conv.lif1.qtrick': tensor(0.5194, device='cuda:0'), 'model.model.6.1.Conv.lif2.qtrick': tensor(0.4493, device='cuda:0'), 'model.model.6.1.Conv.lif3.qtrick': tensor(0.3625, device='cuda:0'), 'model.model.6.1.lif1.qtrick': tensor(0.5300, device='cuda:0'), 'model.model.6.1.lif2.qtrick': tensor(0.1875, device='cuda:0'), 'model.model.6.2.Conv.lif1.qtrick': tensor(0.4678, device='cuda:0'), 'model.model.6.2.Conv.lif2.qtrick': tensor(0.4403, device='cuda:0'), 'model.model.6.2.Conv.lif3.qtrick': tensor(0.3736, device='cuda:0'), 'model.model.6.2.lif1.qtrick': tensor(0.5173, device='cuda:0'), 'model.model.6.2.lif2.qtrick': tensor(0.2694, device='cuda:0'), 'model.model.7.encode_lif.qtrick': tensor(0.6120, device='cuda:0'), 'model.model.8.Conv.lif1.qtrick': tensor(0.2131, device='cuda:0'), 'model.model.8.Conv.lif2.qtrick': tensor(0.4809, device='cuda:0'), 'model.model.8.Conv.lif3.qtrick': tensor(0.5531, device='cuda:0'), 'model.model.8.lif1.qtrick': tensor(0.3559, device='cuda:0'), 'model.model.8.lif2.qtrick': tensor(0.6605, device='cuda:0'), 'model.model.9.cv1.lif.qtrick': tensor(0.3365, device='cuda:0'), 'model.model.9.cv2.lif.qtrick': tensor(3.2746, device='cuda:0'), 'model.model.10.lif.qtrick': tensor(0.5832, device='cuda:0'), 'model.model.12.Conv.lif1.qtrick': tensor(0.4407, device='cuda:0'), 'model.model.12.Conv.lif2.qtrick': tensor(0.8306, device='cuda:0'), 'model.model.12.Conv.lif3.qtrick': tensor(0.5667, device='cuda:0'), 'model.model.12.lif1.qtrick': tensor(0.6396, device='cuda:0'), 'model.model.12.lif2.qtrick': tensor(0.6231, device='cuda:0'), 'model.model.14.lif.qtrick': tensor(0.6123, device='cuda:0'), 'model.model.16.Conv.lif1.qtrick': tensor(0.9470, device='cuda:0'), 'model.model.16.Conv.lif2.qtrick': tensor(1.0075, device='cuda:0'), 'model.model.16.Conv.lif3.qtrick': tensor(0.5116, device='cuda:0'), 'model.model.16.conv1.lif.qtrick': tensor(1.0954, device='cuda:0'), 'model.model.16.conv2.lif.qtrick': tensor(0.1425, device='cuda:0'), 'model.model.18.lif.qtrick': tensor(1.5290, device='cuda:0'), 'model.model.19.Conv.lif1.qtrick': tensor(1.1004, device='cuda:0'), 'model.model.19.Conv.lif2.qtrick': tensor(0.9018, device='cuda:0'), 'model.model.19.Conv.lif3.qtrick': tensor(0.7095, device='cuda:0'), 'model.model.19.conv1.lif.qtrick': tensor(1.3217, device='cuda:0'), 'model.model.19.conv2.lif.qtrick': tensor(0.1250, device='cuda:0'), 'model.model.20.lif.qtrick': tensor(1.6239, device='cuda:0'), 'model.model.22.Conv.lif1.qtrick': tensor(0.8359, device='cuda:0'), 'model.model.22.Conv.lif2.qtrick': tensor(0.5512, device='cuda:0'), 'model.model.22.Conv.lif3.qtrick': tensor(0.5139, device='cuda:0'), 'model.model.22.lif1.qtrick': tensor(0.7395, device='cuda:0'), 'model.model.22.lif2.qtrick': tensor(0.2991, device='cuda:0'), 'model.model.23.lif.qtrick': tensor(0.8043, device='cuda:0'), 'model.model.25.Conv.lif1.qtrick': tensor(0.4260, device='cuda:0'), 'model.model.25.Conv.lif2.qtrick': tensor(0.5488, device='cuda:0'), 'model.model.25.Conv.lif3.qtrick': tensor(0.4577, device='cuda:0'), 'model.model.25.lif1.qtrick': tensor(0.4325, device='cuda:0'), 'model.model.25.lif2.qtrick': tensor(0.5454, device='cuda:0'), 'model.model.26.cv2.0.0.lif.qtrick': tensor(1.6239, device='cuda:0'), 'model.model.26.cv2.0.1.lif.qtrick': tensor(1.5528, device='cuda:0'), 'model.model.26.cv2.0.2.lif.qtrick': tensor(2.8561, device='cuda:0'), 'model.model.26.cv3.0.0.lif.qtrick': tensor(1.6239, device='cuda:0'), 'model.model.26.cv3.0.1.lif.qtrick': tensor(0.6264, device='cuda:0'), 'model.model.26.cv3.0.2.lif.qtrick': tensor(2.5451, device='cuda:0'), 'model.model.26.cv2.1.0.lif.qtrick': tensor(0.8043, device='cuda:0'), 'model.model.26.cv2.1.1.lif.qtrick': tensor(1.5988, device='cuda:0'), 'model.model.26.cv2.1.2.lif.qtrick': tensor(3.4790, device='cuda:0'), 'model.model.26.cv3.1.0.lif.qtrick': tensor(0.8043, device='cuda:0'), 'model.model.26.cv3.1.1.lif.qtrick': tensor(0.6128, device='cuda:0'), 'model.model.26.cv3.1.2.lif.qtrick': tensor(2.5303, device='cuda:0'), 'model.model.26.cv2.2.0.lif.qtrick': tensor(0.5247, device='cuda:0'), 'model.model.26.cv2.2.1.lif.qtrick': tensor(1.4922, device='cuda:0'), 'model.model.26.cv2.2.2.lif.qtrick': tensor(3.7612, device='cuda:0'), 'model.model.26.cv3.2.0.lif.qtrick': tensor(0.5247, device='cuda:0'), 'model.model.26.cv3.2.1.lif.qtrick': tensor(0.6861, device='cuda:0'), 'model.model.26.cv3.2.2.lif.qtrick': tensor(2.4751, device='cuda:0')}
HeinZingerRyo commented 1 day ago

I've converted coco format annotations to yolo format, and put them in my_coco_dir/labels, and the evaluation results with yolo are as below: 微信图片_20241112165740

It's slightly different from evaluation results with coco json annotations, but is still about 4% mAP lower than the reported 66.2% mAP.

HeinZingerRyo commented 1 day ago

Sry I got it, I forgot to check the model size. It seems that this result refers to the 23.1M model result in Table 1. Kindly provide those pretrained model weights with larger parameter numbers, if available?