Problem with test.py

[razvan-tanase]

Hello,

I used the script test.py on a custom dataset with the labels in original YOLOv5 format: < object-class-ID > < X center > < Y center > < Box width > < Box height >

This is the result: Scanning 'C:\Users\rtanase\PycharmProjects\pythonProject\datasets\test\set\labels.cache' for images and labels... 1727 Class: all
Images: 1.73e+03 Targets: 3.61e+03
P: 1 R: 1
mAP@.5: 1 mAP@.5:.95: 1 100%|█| 108/108 [05:58<00 Speed: 19.8/2.0/21.8 ms inference/NMS/total per 640x640 image at batch-size 32 Results saved to runs\test\exp6

Which I got with the following command: python test.py --weights yolov7.pt --data .\data\face_train.yaml --verbose --save-txt --save-hybrid --save-conf

And this is how the face_train.yaml looks like:

val: path/to/images (this is not the real one)

# number of classes
nc: 1

# class names
names: [ 'face' ]

The problem with this result is that it is obviously absolutely impossible for the model to have 100% detection accuracy and produce exactly the same bounding boxes as the ground truth ones.

Furthermore, I tested and confirmed the following:

1. The labels in the labels folder produced after testing with test.py are not the ones returned by the model after detection, but are the ground truths provided with the images to be tested.
2. The coordinates of the labels in the labels folder produced after testing with test.py are not in the standard YOLOv5 format (the one I described above), but describe 2 points of the polygon representing the bounding box.
3. In the mosaic images, created after testing with test.py, are not the bounding boxes resulting from the detection, but the ground truths provided with the images to be tested.

To get these results, I used the debugger to trace the code and slipped a few (intentionally placed) wrong labels along with the images to be tested to observe the result.

My question is whether the script can also be used to validate images from a custom dataset (as in my case) or is it just an internal tool to demonstrate the performance of the model on the COCO dataset, on which it was originally trained.

TL;DR Can test.py also be used on a custom dataset?

[HenrikBradland-Nor]

Remove --save-hybrid

[vijjisomu]

Hi @razvan-tanase Can you please tell me if you were able to solve the issue .I am also facing the same issue.

[vijjisomu]

root@blue-Z390-M:/mnt/fileserver3/Vijayalakshmi/yolov7-main# python test.py --data data/coco-test.yaml --img 640 --batch 32 --conf 0.001 --iou 0.65 --device 0 --weights yolov7.pt --name yolov7_640_val Namespace(augment=False, batch_size=32, conf_thres=0.001, data='data/coco-test.yaml', device='0', exist_ok=False, img_size=640, iou_thres=0.65, name='yolov7_640_val', no_trace=False, project='runs/test', save_conf=False, save_hybrid=False, save_json=False, save_txt=False, single_cls=False, task='val', v5_metric=False, verbose=False, weights=['yolov7.pt']) YOLOR 🚀 2022-10-8 torch 1.10.0a0+3fd9dcf CUDA:0 (NVIDIA GeForce RTX 2080 Ti, 11016.3125MB)

Fusing layers... RepConv.fuse_repvgg_block RepConv.fuse_repvgg_block RepConv.fuse_repvgg_block Model Summary: 306 layers, 36905341 parameters, 36905341 gradients, 104.5 GFLOPS Convert model to Traced-model... /opt/conda/lib/python3.8/site-packages/torch/_tensor.py:987: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/pytorch/pytorch/build/aten/src/ATen/core/TensorBody.h:797.) return self._grad traced_script_module saved! model is traced!

val: Scanning 'data/cocoTest1.cache' images and labels... 4 found, 0 missing, 0 Class Images Labels P R mAP@.5[3 2 1 0] tensor([ 0, 0, 0, 0, 32, 34, 35, 35], device='cuda:0', dtype=torch.int32) Class Images Labels P R mAP@.5 Traceback (most recent call last): File "test.py", line 319, in test(opt.data, File "test.py", line 188, in test confusion_matrix.process_batch(predn, torch.cat((labels[:, 0:1], tbox), 1)) File "/mnt/fileserver3/Vijayalakshmi/yolov7-main/utils/metrics.py", line 160, in process_batch self.matrix[dc, self.nc] += 1 # background FN IndexError: index 32 is out of bounds for axis 0 with size 15

I am stuck with the above issue while running test.py on custom dataset

[mehran66]

--save-hybrid leads to MAP of 1 that is a bug!

[cloudlakecho]

@razvan-tanase Can test.py also be used on a custom dataset?
  Yes
  As usual, you need different pre trained neural network weight. Please train with the custom dataset first.

  Here are files to modify
   YAML file in "data" folder (if you have different label(s), please modify "names")
    Related TXT file in the YAML file ("train" and "val")

[ainiazmi]

Remove --save-hybrid

Can you share how to remove --save-hybrid.

[ainiazmi]

root@blue-Z390-M:/mnt/fileserver3/Vijayalakshmi/yolov7-main# python test.py --data data/coco-test.yaml --img 640 --batch 32 --conf 0.001 --iou 0.65 --device 0 --weights yolov7.pt --name yolov7_640_val Namespace(augment=False, batch_size=32, conf_thres=0.001, data='data/coco-test.yaml', device='0', exist_ok=False, img_size=640, iou_thres=0.65, name='yolov7_640_val', no_trace=False, project='runs/test', save_conf=False, save_hybrid=False, save_json=False, save_txt=False, single_cls=False, task='val', v5_metric=False, verbose=False, weights=['yolov7.pt']) YOLOR 🚀 2022-10-8 torch 1.10.0a0+3fd9dcf CUDA:0 (NVIDIA GeForce RTX 2080 Ti, 11016.3125MB)

Fusing layers... RepConv.fuse_repvgg_block RepConv.fuse_repvgg_block RepConv.fuse_repvgg_block Model Summary: 306 layers, 36905341 parameters, 36905341 gradients, 104.5 GFLOPS Convert model to Traced-model... /opt/conda/lib/python3.8/site-packages/torch/_tensor.py:987: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at /opt/pytorch/pytorch/build/aten/src/ATen/core/TensorBody.h:797.) return self._grad traced_script_module saved! model is traced!

val: Scanning 'data/cocoTest1.cache' images and labels... 4 found, 0 missing, 0 Class Images Labels P R mAP@.5[3 2 1 0] tensor([ 0, 0, 0, 0, 32, 34, 35, 35], device='cuda:0', dtype=torch.int32) Class Images Labels P R mAP@.5 Traceback (most recent call last): File "test.py", line 319, in test(opt.data, File "test.py", line 188, in test confusion_matrix.process_batch(predn, torch.cat((labels[:, 0:1], tbox), 1)) File "/mnt/fileserver3/Vijayalakshmi/yolov7-main/utils/metrics.py", line 160, in process_batch self.matrix[dc, self.nc] += 1 # background FN IndexError: index 32 is out of bounds for axis 0 with size 15

I am stuck with the above issue while running test.py on custom dataset

did you able to solve this problem. I also faced this issue when I try to run the test.py using coco dataset. Can you share how you solve this problem.

[hashimotol]

Reference

did you ever figure it out. Im facing the same issue