how to validate on test data set instead of val data set

Shuixin-Li commented 1 year ago

Before Asking

[X] I have read the README carefully. 我已经仔细阅读了README上的操作指引。
[X] I want to train my custom dataset, and I have read the tutorials for training your custom data carefully and organize my dataset correctly; (FYI: We recommand you to apply the config files of xx_finetune.py.) 我想训练自定义数据集，我已经仔细阅读了训练自定义数据的教程，以及按照正确的目录结构存放数据集。（FYI: 我们推荐使用xx_finetune.py等配置文件训练自定义数据集。）
[X] I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking

[X] I have searched the YOLOv6 issues and found no similar questions.

Question

I split my data into train, val, test as usual, I sucessfully evaluated on val data set after training, but I received error when I try to evaluate test dataset.

steps

create a coco format annotation json file for test dataset name instances_test.json I think I may need to create instances_test.json somewhere instead of creating them.
run the command python tools/eval.py --data data/dataset.yaml --weights runs/train/exp/weights/best_ckpt.pt --task test --device 0

log

nohup: ignoring input
Namespace(batch_size=32, conf_thres=0.03, config_file='', data='data/dataset.yaml', device='0', do_coco_metric=True, do_pr_metric=False, eval_config_file='./configs/experiment/eval_640_repro.py', half=False, height=None, img_size=640, infer_on_rect=True, iou_thres=0.65, name='exp', plot_confusion_matrix=False, plot_curve=True, reproduce_640_eval=False, save_dir='runs/val/', shrink_size=0, specific_shape=False, task='test', verbose=False, weights='runs/train/exp/weights/best_ckpt.pt', width=None)
Loading checkpoint from runs/train/exp/weights/best_ckpt.pt

Fusing model...
Switch model to deploy modality.
Model Summary: Params: 18.54M, Gflops: 45.28
img record infomation path is:../datasets/yolov6/images/.test_cache.json
Test: Checking formats of labels with 8 process(es): 

  0%|          | 0/217 [00:00<?, ?it/s]
217 label(s) found, 0 label(s) missing, 21 label(s) empty, 0 invalid label files: 100%|██████████| 217/217 [00:00<00:00, 18256.59it/s]
Test: Final numbers of valid images: 217/ labels: 217. 
0.4s for dataset initialization.

Inferencing model in test datasets.:   0%|                | 0/7 [00:00<?, ?it/s]
Inferencing model in test datasets.:  14%|█▏      | 1/7 [00:01<00:08,  1.43s/it]
Inferencing model in test datasets.:  29%|██▎     | 2/7 [00:02<00:05,  1.16s/it]
Inferencing model in test datasets.:  43%|███▍    | 3/7 [00:03<00:03,  1.05it/s]
Inferencing model in test datasets.:  57%|████▌   | 4/7 [00:03<00:02,  1.08it/s]
Inferencing model in test datasets.:  71%|█████▋  | 5/7 [00:04<00:01,  1.13it/s]
Inferencing model in test datasets.:  86%|██████▊ | 6/7 [00:05<00:00,  1.10it/s]
Inferencing model in test datasets.: 100%|████████| 7/7 [00:06<00:00,  1.21it/s]
Inferencing model in test datasets.: 100%|████████| 7/7 [00:06<00:00,  1.09it/s]

Evaluating speed.
Average pre-process time: 0.30 ms
Average inference time: 8.26 ms
Average NMS time: 4.33 ms

Evaluating mAP by pycocotools.
Saving runs/val/exp5/predictions.json...
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Loading and preparing results...
Traceback (most recent call last):
  File "tools/eval.py", line 169, in <module>
    main(args)
  File "tools/eval.py", line 164, in main
    run(**vars(args))
  File "~/anaconda3/envs/yolov6/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "tools/eval.py", line 159, in run
    eval_result = val.eval_model(pred_result, model, dataloader, task)
  File "~/Documents/YOLOv6/yolov6/core/evaler.py", line 260, in eval_model
    pred = anno.loadRes(pred_json)
  File "~/anaconda3/envs/yolov6/lib/python3.8/site-packages/pycocotools/coco.py", line 327, in loadRes
    assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \
AssertionError: Results do not correspond to current coco set

dataset.yaml

# Please insure that your custom_dataset are put in same parent dir with YOLOv6_DIR
train: ../datasets/yolov6/images/train # train images
val: ../datasets/yolov6/images/valid # val images
test: ../datasets/yolov6/images/test # test images (optional)

# whether it is coco dataset, only coco dataset should be set to True.
is_coco: False
# Classes
nc: 16  # number of classes
names: [...]  # class names

Additional

No response

Shuixin-Li commented 1 year ago

@mtjhl maybe you can help with this question?

mtjhl commented 1 year ago

For easy, you can set test path in dataset yaml file to val: ， for example

train: ../coco/images/train2017 # 118287 images
# 把 val path 设置为 test 的path
val: ../coco/images/val2017  # 5000 images 
test: ../coco/images/test2017

mtjhl commented 1 year ago

You do not need to create instance_test.json by your self， yolov6 will generate it automatically according to yolo format labels.

duzongdong commented 4 months ago

为方便起见，您可以将数据集 yaml 文件中的测试路径设置为，例如val:
train: ../coco/images/train2017 # 118287 images
# 把 val path 设置为 test 的path
val: ../coco/images/val2017  # 5000 images 
test: ../coco/images/test2017
但是这样还是对val数据集进行验证，我的数据集也是划分了3部分，包含test数据集。有什么方法生成test数据集对应的json文件，就像生成Val的json文件一样。来达到对test数据集进行验证的目的。

duzongdong commented 4 months ago

为方便起见，您可以将数据集 yaml 文件中的测试路径设置为，例如val:
train: ../coco/images/train2017 # 118287 images
# 把 val path 设置为 test 的path
val: ../coco/images/val2017  # 5000 images 
test: ../coco/images/test2017
但是这样还是对val数据集进行验证，我的数据集也是划分了3部分，包含test数据集。有什么方法生成test数据集对应的json文件，就像生成Val的json文件一样。来达到对test数据集进行验证的目的。

我明白了，数据集分为三个部分，第一次训练使用val数据集进行训练中的验证。训练结束后将数据集配置文件中的val:路径改为test的路径，然后再train.py一次即可以生成instances_test.json，训练中断即可;主要为了得到json文件然后使用eval.py进行test数据集的验证。这样数据集三个部分都用到了。

meituan / YOLOv6