set config with no val and test annotation 训练自己数据集没有测试集标注和验证集，但是config似乎必须要求填?

[HaoGuo98]

i want to train and test my coco style dataset. but i do not have the val set and test annotation. However, in the dataset config, it seems that the key ann_file is necessary. So i fill it with train annotation in val and test. Now i successfully train my model(maybe?) But when i test my model, it use the train images not the test images. I wander if it is because i fill the test annotation by train annotation?

[ElectronicElephant]

I wander if it is because i fill the test annotation by train annotation?

Yes. You have to split the dataset into train, test, and val. Otherwise, you basically have NO way to decide if a model is good or not.

[HaoGuo98]

I wander if it is because i fill the test annotation by train annotation?

Yes. You have to split the dataset into train, test, and val. Otherwise, you basically have NO way to decide if a model is good or not.

the task do not provide annotation of test set, and the train set is quite small, so i do not want split it into val set. But the config require the ann_file of val and test, it make me confused. Or it would be helpful if you can show me how to test the the test set of coco 2017. In the default config, the test set of coco use ann_file and image_prefix like the val set. But the test set of coco2017 also do not have the ann_file.

[ElectronicElephant]

I wander if it is because i fill the test annotation by train annotation?

Yes. You have to split the dataset into train, test, and val. Otherwise, you basically have NO way to decide if a model is good or not.

the task do not provide annotation of test set, and the train set is quite small, so i do not want split it into val set. But the config require the ann_file of val and test, it make me confused. Or it would be helpful if you can show me how to test the the test set of coco 2017. In the default config, the test set of coco use ann_file and image_prefix like the val set. But the test set of coco2017 also do not have the ann_file.

Then, IMHO, you should use test.py and --show-dir option to visualize the results on test set.

[HaoGuo98]

I wander if it is because i fill the test annotation by train annotation?

Yes. You have to split the dataset into train, test, and val. Otherwise, you basically have NO way to decide if a model is good or not.

the task do not provide annotation of test set, and the train set is quite small, so i do not want split it into val set. But the config require the ann_file of val and test, it make me confused. Or it would be helpful if you can show me how to test the the test set of coco 2017. In the default config, the test set of coco use ann_file and image_prefix like the val set. But the test set of coco2017 also do not have the ann_file.

Then, IMHO, you should use test.py and --show-dir option to visualize the results on test set.

Yes, i find the test was done on the test set, but it just test the size of train set. I guess it is because i set the ann_file of test set is train set (train.json? But actually i do not have the annotation of test set. Can you show me how to test coco2017 on test set? i think it would be helpful.

[ElectronicElephant]

I wander if it is because i fill the test annotation by train annotation?

Yes. You have to split the dataset into train, test, and val. Otherwise, you basically have NO way to decide if a model is good or not.

the task do not provide annotation of test set, and the train set is quite small, so i do not want split it into val set. But the config require the ann_file of val and test, it make me confused. Or it would be helpful if you can show me how to test the the test set of coco 2017. In the default config, the test set of coco use ann_file and image_prefix like the val set. But the test set of coco2017 also do not have the ann_file.

Then, IMHO, you should use test.py and --show-dir option to visualize the results on test set.

Yes, i find the test was done on the test set, but it just test the size of train set. I guess it is because i set the ann_file of test set is train set (train.json? But actually i do not have the annotation of test set.

Hi, you can try this https://mmdetection.readthedocs.io/en/latest/1_exist_data_model.html#image-demo. It can visualize the predicted results given image path. -:)

[songqi-github]

When visualizing the results as the given link, the ann_file will also limit the number of images to be predicted. Could you please directly tell me how to generate the test results on COCO test set without given any ann_file or ann_file path? How to modify the code to meet this need?

[ElectronicElephant]

Hi, maybe something like this

from mmdet.apis import init_detector, inference_detector, show_result_pyplot

class Detector:
    def __init__(self, config_file, checkpoint_file, device='cpu') -> None:
        self.config_file = config_file
        self.checkpoint_file = checkpoint_file
        self.device = device

        self.model = init_detector(config_file, checkpoint_file, device=device)

    def inference(self, img, save_path):
        # img: rgb or path to image
        result = inference_detector(self.model, img)
        result_img = self.model.show_result(img, result, score_thr=0.5)
        cv2.imwrite(save_path, result_img)

### Main
config_file = 'models/fcos_coco_randbg8000_0.1_0.5.py'
checkpoint_file = 'models/fcos_coco_randbg8000_0.1_0.5.pth'
device = 'cuda:0'
det = Detector(config_file, checkpoint_file, device) 

for img in glob.glob('xxxxx/xxxxx/*.jpg'):
    save_path = xxxxx
    det.inference(img, save_path)

BTW, I don't think demo.py relies the anno file, cuz the path to img file is given by the command line.

[songqi-github]

Hi, I think you misunderstand my needing and question. I used the following command to generate the test results on COCO test set, which should be then uploaded to COCO's testing website to do the accuracy testing.

./tools/dist_test.sh \ configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py \ checkpoints/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth \ 8 \ --format-only \ --options "jsonfile_prefix=./mask_rcnn_test-dev_results"

In this situation, the ann_file of COCO test set is None, but if we set it to None, there will raise some mistakes. And if we set the ann_file path the same as the val set, then the predicting json only contains the same number of results as val json.

[ElectronicElephant]

Hi, I think you misunderstand my needing and question. I used the following command to generate the test results on COCO test set, which should be then uploaded to COCO's testing website to do the accuracy testing.

./tools/dist_test.sh \ configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py \ checkpoints/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth \ 8 \ --format-only \ --options "jsonfile_prefix=./mask_rcnn_test-dev_results"

In this situation, the ann_file of COCO test set is None, but if we set it to None, there will raise some mistakes. And if we set the ann_file path the same as the val set, then the predicting json only contains the same number of results as val json.

emmm, I originally thought you tried to test on your customized dataset -:)

The image info of COCO 2017 can be downloaded here. The extracted JSON file could serve as the anno in config. http://images.cocodataset.org/annotations/image_info_test2017.zip

[songqi-github]

The extracted JSON file is used as a placeholder? If I changed the COCO dataset to another one which also doesn't contain the ann_file for the test set, how should I generate the test results? It seems weird to extract JSON files as anno.

[ElectronicElephant]

The extracted JSON file is used as a placeholder? If I changed the COCO dataset to another one which also doesn't contain the ann_file for the test set, how should I generate the test results? It seems weird to extract JSON files as anno.

In COCO format, there is an img_id for each of the image, no matter it's in train set or test set. Also, there is an anno_id for each annotation. Thus, these two json files is to provide img_id for each image. Then, each of your prediction can be in this format:

{'img_id': xxxx, 'bbox': xxxxx, 'segm':[ xxx, xxxx]}

Even if you're using your own dataset, you should generate these json files for test set (they contains img_ids but no annotations). Details can be referred to https://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch

[songqi-github]

Could you please give me the file used to generate these json files for test set (they contains img_ids but no annotations). THX.

[songqi-github]

I didn't find that in the link you given.

[ElectronicElephant]

http://images.cocodataset.org/annotations/image_info_test2017.zip

[songqi-github]

emmm, I mean the python file used to generate these JSON files.

[ElectronicElephant]

emmm, I mean the python file used to generate these JSON files.

Hi, you may refer to https://github.com/waspinator/pycococreator. In my situations, it always work like a charm.

[songqi-github]

OK, thank you very much.