[Bug] Accuracy-Check: It does not support the annotation with the COCO text files but it works on JSON format

[qin53027]

Hi,

I try to run the accuracy check on person-detection-retail-0002 model with JSON annotation file. But get bad result:

1. using the COCO2017 json file: instances_val2017.json ===> got AP = 0.01%

2. I generated the filtered_instances_val2017.json with only Person class ====> got AP = 0.00%

3. I extracted the Person class from val2017 and got the person_images and person_annotation with txt files When I use following command to run:

accuracy_check -c /home/jiande/program/open_model_zoo/tools/accuracy_checker/configs/person-detection-retail-0002.yml -m /tmp/model -s /home/jiande/coco_kitti -a /home/jiande/coco_kitti/labels

then I got error: File "/home/jiande/.local/lib/python3.8/site-packages/accuracy_checker/dataset.py", line 153, in load_annotation raise ConfigError('path to converted annotation or data for conversion should be specified') accuracy_checker.config.config_validator.ConfigError: path to converted annotation or data for conversion should be specified

Please let me know how I can use the annotation with the text format. In this way, I can compare the AP result. Also if I can use the annotation with text file format, please let me know how I can include the person_images I extracted.

Thanks so much Jimmy

run person-detection-retail-0002 command

accuracy_check -m /tmp/model/intel/person-detection-retail-0002/FP32 -c /home/jiande/coco_kitti/person-detection-retail-0002.yml

AP result ---- only 0.01%

Output info Layer name: detection_out precision: FP32 shape: [1, 1, 200, 7]

5000 objects processed in 753.631 seconds map: 0.01%

person-detection-retail-0002.yml

models:

• name: person-detection-retail-0002

  launchers:

  • framework: dlsdk device: CPU adapter: ssd inputs:
    • name: im_info type: CONST_INPUT value: [[544, 992, 0, 0, 0, 0]]

  datasets:

  • name: person_detection data_source: /home/jiande/coco_kitti/val2017 annotation_conversion: converter: mscoco_detection annotation_file: /home/jiande/coco_kitti/annotations/filtered_instances_val2017.json reader: pillow_imread

    preprocessing: models:

• name: person-detection-retail-0002

  launchers:

  • framework: dlsdk device: CPU adapter: ssd inputs:
    • name: im_info type: CONST_INPUT value: [[544, 992, 0, 0, 0, 0]]

  datasets:

  • name: person_detection data_source: /home/jiande/coco_kitti/val2017 annotation_conversion: converter: mscoco_detection annotation_file: /home/jiande/coco_kitti/annotations/filtered_instances_val2017.json reader: pillow_imread

    preprocessing:

    • type: resize dst_width: 992 dst_height: 544

    postprocessing:

    • type: resize_prediction_boxes
    • type: filter apply_to: prediction min_confidence: 0.0 width_range: 5 height_range: 5
    • type: cast_to_int
    • type: clip_boxes apply_to: annotation dst_width: 1920 dst_height: 1080
    • type: filter apply_to: annotation height_range: 100, 10000 is_empty: True min_visibility: partially occluded
    • type: clip_boxes apply_to: prediction dst_width: 1920 dst_height: 1080
    • type: filter apply_to: prediction height_range: 100 is_empty: True aspect_ratio: 0.666, 5

    metrics:

    • type: map ignore_difficult: True include_boundaries: False allow_multiple_matches_per_ignored: False distinct_conf: False

[zulkifli-halim]

Hi @qin53027, the COCO dataset might not work as it's targeted more for object detection. We suggest you use this Pedestrian Detection Database: https://www.cis.upenn.edu/~jshi/ped_html/

[qin53027]

let me close this issue first