search

facebookresearch / detectron2

Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.
https://detectron2.readthedocs.io/en/latest/
Apache License 2.0
30.69k stars 7.51k forks source link

Low AP values in the evaluation matrix #1615

Closed dxlong2000 closed 4 years ago

dxlong2000 commented 4 years ago

Dear all,

I try to evaluate the Citispace model running on the val2017 dataset from Coco dataset website and I got the matrix as follow:

  1. full code you wrote or full changes you made (git diff) 
    
Evaluation results for segm: 
|  AP   |  AP50  |  AP75  |  APs  |  APm  |  APl  |
|:-----:|:------:|:------:|:-----:|:-----:|:-----:|
| 0.562 | 1.005  | 0.568  | 0.371 | 0.779 | 0.831 |
Per-category segm AP: 
| category      | AP     | category     | AP    | category       | AP     |
|:--------------|:-------|:-------------|:------|:---------------|:-------|
| person        | 20.615 | bicycle      | 0.000 | car            | 24.235 |
| motorcycle    | 0.000  | airplane     | 0.000 | bus            | 0.107  |
| train         | 0.000  | truck        | 0.000 | boat           | 0.000  |
| traffic light | 0.000  | fire hydrant | 0.000 | stop sign      | 0.000  |
| parking meter | 0.000  | bench        | 0.000 | bird           | 0.000  |
| cat           | 0.000  | dog          | 0.000 | horse          | 0.000  |
| sheep         | 0.000  | cow          | 0.000 | elephant       | 0.000  |
| bear          | 0.000  | zebra        | 0.000 | giraffe        | 0.000  |
| backpack      | 0.000  | umbrella     | 0.000 | handbag        | 0.000  |
| tie           | 0.000  | suitcase     | 0.000 | frisbee        | 0.000  |
| skis          | 0.000  | snowboard    | 0.000 | sports ball    | 0.000  |
| kite          | 0.000  | baseball bat | 0.000 | baseball glove | 0.000  |
| skateboard    | 0.000  | surfboard    | 0.000 | tennis racket  | 0.000  |
| bottle        | 0.000  | wine glass   | 0.000 | cup            | 0.000  |
| fork          | 0.000  | knife        | 0.000 | spoon          | 0.000  |
| bowl          | 0.000  | banana       | 0.000 | apple          | 0.000  |
| sandwich      | 0.000  | orange       | 0.000 | broccoli       | 0.000  |
| carrot        | 0.000  | hot dog      | 0.000 | pizza          | 0.000  |
| donut         | 0.000  | cake         | 0.000 | chair          | 0.000  |
| couch         | 0.000  | potted plant | 0.000 | bed            | 0.000  |
| dining table  | 0.000  | toilet       | 0.000 | tv             | 0.000  |
| laptop        | 0.000  | mouse        | 0.000 | remote         | 0.000  |
| keyboard      | 0.000  | cell phone   | 0.000 | microwave      | 0.000  |
| oven          | 0.000  | toaster      | 0.000 | sink           | 0.000  |
| refrigerator  | 0.000  | book         | 0.000 | clock          | 0.000  |
| vase          | 0.000  | scissors     | 0.000 | teddy bear     | 0.000  |
| hair drier    | 0.000  | toothbrush   | 0.000 |                |        |```

It seems that all the evaluation values are low. Does anyone give me any suggestions about this low?

Thanks in advanced!

ppwwyyxx commented 4 years ago

If you need help to solve an unexpected issue you observed, please include details following the "Unexpected behaviors" issue template.

dxlong2000 commented 4 years ago

Dear Mr. Yuxin Wu Problem: Low AP values in the evaluation matrices. Data I used from COCO Dataset val2017.

!wget http://images.cocodataset.org/zips/val2017.zip
!unzip val2017.zip > /dev/null

Instructions To Reproduce the Issue:

  1. full code you wrote or full changes you made (git diff) 
    
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("Cityscapes/mask_rcnn_R_50_FPN.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
# Find a model from detectron2's model zoo. You can use the https://dl.fbaipublicfiles... url as well
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("Cityscapes/mask_rcnn_R_50_FPN.yaml")

from detectron2.data.datasets import register_coco_instances register_cocoinstances("val2017", {}, "val2017/"+"instances_val2017.json", "val2017") datasetdicts = DatasetCatalog.get("val2017") val2017metadata = MetadataCatalog.get("val2017")

model = build_model(cfg) from detectron2.checkpoint import DetectionCheckpointer DetectionCheckpointer(model).load(cfg.MODEL.WEIGHTS)
checkpointer = DetectionCheckpointer(model, save_dir="output") checkpointer.save("model_99")

from detectron2.evaluation import COCOEvaluator, inference_on_dataset from detectron2.data import build_detection_testloader evaluator = COCOEvaluator("val2017", cfg, True, output_dir="./output/") val_loader = build_detection_testloader(cfg, "val2017") inference_on_dataset(model, val_loader, evaluator)

2. what exact command you run: I run on Google Colab
3. __full logs__ you observed:

WARNING [06/18 01:20:22 d2.data.datasets.coco]: Category ids in annotations are not in [1, #categories]! We'll apply a mapping for you.

[06/18 01:20:22 d2.data.datasets.coco]: Loaded 5000 images in COCO format from val2017/instances_val2017.json [06/18 01:20:22 d2.data.build]: Distribution of instances among all 80 categories: category #instances category #instances category #instances
person 10777 bicycle 314 car 1918
motorcycle 367 airplane 143 bus 283
train 190 truck 414 boat 424
traffic light 634 fire hydrant 101 stop sign 75
parking meter 60 bench 411 bird 427
cat 202 dog 218 horse 272
sheep 354 cow 372 elephant 252
bear 71 zebra 266 giraffe 232
backpack 371 umbrella 407 handbag 540
tie 252 suitcase 299 frisbee 115
skis 241 snowboard 69 sports ball 260
kite 327 baseball bat 145 baseball gl.. 148
skateboard 179 surfboard 267 tennis racket 225
bottle 1013 wine glass 341 cup 895
fork 215 knife 325 spoon 253
bowl 623 banana 370 apple 236
sandwich 177 orange 285 broccoli 312
carrot 365 hot dog 125 pizza 284
donut 328 cake 310 chair 1771
couch 261 potted plant 342 bed 163
dining table 695 toilet 179 tv 288
laptop 231 mouse 106 remote 283
keyboard 153 cell phone 262 microwave 55
oven 143 toaster 9 sink 225
refrigerator 126 book 1129 clock 267
vase 274 scissors 36 teddy bear 190
hair drier 11 toothbrush 57
total 36335
[06/18 01:20:22 d2.data.common]: Serializing 5000 elements to byte tensors and concatenating them all ... [06/18 01:20:22 d2.data.common]: Serialized dataset takes 19.33 MiB [06/18 01:20:22 d2.evaluation.evaluator]: Start inference on 5000 images [06/18 01:20:25 d2.evaluation.evaluator]: Inference done 11/5000. 0.2003 s / img. ETA=0:17:02 [06/18 01:20:30 d2.evaluation.evaluator]: Inference done 35/5000. 0.2044 s / img. ETA=0:17:11 [06/18 01:20:35 d2.evaluation.evaluator]: Inference done 59/5000. 0.2072 s / img. ETA=0:17:20 [06/18 01:20:40 d2.evaluation.evaluator]: Inference done 83/5000. 0.2080 s / img. ETA=0:17:19 [06/18 01:20:45 d2.evaluation.evaluator]: Inference done 109/5000. 0.2049 s / img. ETA=0:16:57 [06/18 01:20:50 d2.evaluation.evaluator]: Inference done 133/5000. 0.2055 s / img. ETA=0:16:57 [06/18 01:20:55 d2.evaluation.evaluator]: Inference done 157/5000. 0.2064 s / img. ETA=0:16:56 [06/18 01:21:01 d2.evaluation.evaluator]: Inference done 182/5000. 0.2055 s / img. ETA=0:16:46 [06/18 01:21:06 d2.evaluation.evaluator]: Inference done 206/5000. 0.2057 s / img. ETA=0:16:42 [06/18 01:21:11 d2.evaluation.evaluator]: Inference done 231/5000. 0.2051 s / img. ETA=0:16:34 [06/18 01:21:16 d2.evaluation.evaluator]: Inference done 255/5000. 0.2054 s / img. ETA=0:16:30 [06/18 01:21:21 d2.evaluation.evaluator]: Inference done 280/5000. 0.2052 s / img. ETA=0:16:23 [06/18 01:21:26 d2.evaluation.evaluator]: Inference done 303/5000. 0.2058 s / img. ETA=0:16:22 [06/18 01:21:31 d2.evaluation.evaluator]: Inference done 328/5000. 0.2057 s / img. ETA=0:16:16 [06/18 01:21:36 d2.evaluation.evaluator]: Inference done 352/5000. 0.2061 s / img. ETA=0:16:14 [06/18 01:21:41 d2.evaluation.evaluator]: Inference done 376/5000. 0.2065 s / img. ETA=0:16:10 [06/18 01:21:47 d2.evaluation.evaluator]: Inference done 401/5000. 0.2065 s / img. ETA=0:16:05 [06/18 01:21:52 d2.evaluation.evaluator]: Inference done 425/5000. 0.2067 s / img. ETA=0:16:01 [06/18 01:21:57 d2.evaluation.evaluator]: Inference done 449/5000. 0.2069 s / img. ETA=0:15:57 [06/18 01:22:02 d2.evaluation.evaluator]: Inference done 472/5000. 0.2074 s / img. ETA=0:15:54 [06/18 01:22:07 d2.evaluation.evaluator]: Inference done 496/5000. 0.2075 s / img. ETA=0:15:50 [06/18 01:22:12 d2.evaluation.evaluator]: Inference done 521/5000. 0.2072 s / img. ETA=0:15:43 [06/18 01:22:17 d2.evaluation.evaluator]: Inference done 545/5000. 0.2073 s / img. ETA=0:15:39 [06/18 01:22:22 d2.evaluation.evaluator]: Inference done 569/5000. 0.2073 s / img. ETA=0:15:33 [06/18 01:22:27 d2.evaluation.evaluator]: Inference done 593/5000. 0.2072 s / img. ETA=0:15:28 [06/18 01:22:33 d2.evaluation.evaluator]: Inference done 616/5000. 0.2076 s / img. ETA=0:15:25 [06/18 01:22:38 d2.evaluation.evaluator]: Inference done 640/5000. 0.2076 s / img. ETA=0:15:20 [06/18 01:22:43 d2.evaluation.evaluator]: Inference done 664/5000. 0.2075 s / img. ETA=0:15:14 [06/18 01:22:48 d2.evaluation.evaluator]: Inference done 688/5000. 0.2076 s / img. ETA=0:15:10 [06/18 01:22:53 d2.evaluation.evaluator]: Inference done 712/5000. 0.2077 s / img. ETA=0:15:05 [06/18 01:22:58 d2.evaluation.evaluator]: Inference done 736/5000. 0.2078 s / img. ETA=0:15:01 [06/18 01:23:03 d2.evaluation.evaluator]: Inference done 760/5000. 0.2077 s / img. ETA=0:14:55 [06/18 01:23:08 d2.evaluation.evaluator]: Inference done 784/5000. 0.2079 s / img. ETA=0:14:51 [06/18 01:23:13 d2.evaluation.evaluator]: Inference done 808/5000. 0.2079 s / img. ETA=0:14:46 [06/18 01:23:19 d2.evaluation.evaluator]: Inference done 832/5000. 0.2080 s / img. ETA=0:14:41 [06/18 01:23:24 d2.evaluation.evaluator]: Inference done 856/5000. 0.2080 s / img. ETA=0:14:36 [06/18 01:23:29 d2.evaluation.evaluator]: Inference done 880/5000. 0.2080 s / img. ETA=0:14:31 [06/18 01:23:34 d2.evaluation.evaluator]: Inference done 904/5000. 0.2079 s / img. ETA=0:14:26 [06/18 01:23:39 d2.evaluation.evaluator]: Inference done 929/5000. 0.2078 s / img. ETA=0:14:20 [06/18 01:23:44 d2.evaluation.evaluator]: Inference done 953/5000. 0.2079 s / img. ETA=0:14:15 [06/18 01:23:49 d2.evaluation.evaluator]: Inference done 977/5000. 0.2078 s / img. ETA=0:14:10 [06/18 01:23:54 d2.evaluation.evaluator]: Inference done 1001/5000. 0.2079 s / img. ETA=0:14:05 [06/18 01:23:59 d2.evaluation.evaluator]: Inference done 1026/5000. 0.2078 s / img. ETA=0:13:59 [06/18 01:24:05 d2.evaluation.evaluator]: Inference done 1050/5000. 0.2079 s / img. ETA=0:13:55 [06/18 01:24:10 d2.evaluation.evaluator]: Inference done 1073/5000. 0.2081 s / img. ETA=0:13:50 [06/18 01:24:15 d2.evaluation.evaluator]: Inference done 1098/5000. 0.2080 s / img. ETA=0:13:45 [06/18 01:24:20 d2.evaluation.evaluator]: Inference done 1122/5000. 0.2081 s / img. ETA=0:13:40 [06/18 01:24:25 d2.evaluation.evaluator]: Inference done 1145/5000. 0.2082 s / img. ETA=0:13:36 [06/18 01:24:30 d2.evaluation.evaluator]: Inference done 1169/5000. 0.2083 s / img. ETA=0:13:31 [06/18 01:24:35 d2.evaluation.evaluator]: Inference done 1193/5000. 0.2083 s / img. ETA=0:13:26 [06/18 01:24:40 d2.evaluation.evaluator]: Inference done 1216/5000. 0.2084 s / img. ETA=0:13:22 [06/18 01:24:45 d2.evaluation.evaluator]: Inference done 1240/5000. 0.2084 s / img. ETA=0:13:17 [06/18 01:24:51 d2.evaluation.evaluator]: Inference done 1264/5000. 0.2085 s / img. ETA=0:13:12 [06/18 01:24:56 d2.evaluation.evaluator]: Inference done 1288/5000. 0.2085 s / img. ETA=0:13:07 [06/18 01:25:01 d2.evaluation.evaluator]: Inference done 1312/5000. 0.2084 s / img. ETA=0:13:02 [06/18 01:25:06 d2.evaluation.evaluator]: Inference done 1336/5000. 0.2083 s / img. ETA=0:12:56 [06/18 01:25:11 d2.evaluation.evaluator]: Inference done 1360/5000. 0.2084 s / img. ETA=0:12:51 [06/18 01:25:16 d2.evaluation.evaluator]: Inference done 1384/5000. 0.2084 s / img. ETA=0:12:46 [06/18 01:25:21 d2.evaluation.evaluator]: Inference done 1408/5000. 0.2083 s / img. ETA=0:12:41 [06/18 01:25:26 d2.evaluation.evaluator]: Inference done 1433/5000. 0.2083 s / img. ETA=0:12:35 [06/18 01:25:31 d2.evaluation.evaluator]: Inference done 1457/5000. 0.2082 s / img. ETA=0:12:30 [06/18 01:25:36 d2.evaluation.evaluator]: Inference done 1482/5000. 0.2081 s / img. ETA=0:12:25 [06/18 01:25:41 d2.evaluation.evaluator]: Inference done 1506/5000. 0.2081 s / img. ETA=0:12:19 [06/18 01:25:47 d2.evaluation.evaluator]: Inference done 1531/5000. 0.2081 s / img. ETA=0:12:14 [06/18 01:25:52 d2.evaluation.evaluator]: Inference done 1555/5000. 0.2081 s / img. ETA=0:12:09 [06/18 01:25:57 d2.evaluation.evaluator]: Inference done 1579/5000. 0.2081 s / img. ETA=0:12:04 [06/18 01:26:02 d2.evaluation.evaluator]: Inference done 1603/5000. 0.2082 s / img. ETA=0:11:59 [06/18 01:26:07 d2.evaluation.evaluator]: Inference done 1627/5000. 0.2082 s / img. ETA=0:11:54 [06/18 01:26:12 d2.evaluation.evaluator]: Inference done 1652/5000. 0.2082 s / img. ETA=0:11:49 [06/18 01:26:18 d2.evaluation.evaluator]: Inference done 1676/5000. 0.2082 s / img. ETA=0:11:44 [06/18 01:26:23 d2.evaluation.evaluator]: Inference done 1700/5000. 0.2082 s / img. ETA=0:11:39 [06/18 01:26:28 d2.evaluation.evaluator]: Inference done 1724/5000. 0.2081 s / img. ETA=0:11:33 [06/18 01:26:33 d2.evaluation.evaluator]: Inference done 1748/5000. 0.2082 s / img. ETA=0:11:28 [06/18 01:26:38 d2.evaluation.evaluator]: Inference done 1773/5000. 0.2081 s / img. ETA=0:11:23 [06/18 01:26:43 d2.evaluation.evaluator]: Inference done 1797/5000. 0.2081 s / img. ETA=0:11:18 [06/18 01:26:48 d2.evaluation.evaluator]: Inference done 1821/5000. 0.2081 s / img. ETA=0:11:13 [06/18 01:26:53 d2.evaluation.evaluator]: Inference done 1845/5000. 0.2081 s / img. ETA=0:11:08 [06/18 01:26:58 d2.evaluation.evaluator]: Inference done 1869/5000. 0.2081 s / img. ETA=0:11:02 [06/18 01:27:03 d2.evaluation.evaluator]: Inference done 1893/5000. 0.2081 s / img. ETA=0:10:57 [06/18 01:27:09 d2.evaluation.evaluator]: Inference done 1918/5000. 0.2080 s / img. ETA=0:10:52 [06/18 01:27:14 d2.evaluation.evaluator]: Inference done 1942/5000. 0.2080 s / img. ETA=0:10:47 [06/18 01:27:19 d2.evaluation.evaluator]: Inference done 1966/5000. 0.2081 s / img. ETA=0:10:42 [06/18 01:27:24 d2.evaluation.evaluator]: Inference done 1990/5000. 0.2081 s / img. ETA=0:10:37 [06/18 01:27:29 d2.evaluation.evaluator]: Inference done 2014/5000. 0.2081 s / img. ETA=0:10:32 [06/18 01:27:34 d2.evaluation.evaluator]: Inference done 2039/5000. 0.2081 s / img. ETA=0:10:26 [06/18 01:27:39 d2.evaluation.evaluator]: Inference done 2064/5000. 0.2080 s / img. ETA=0:10:21 [06/18 01:27:45 d2.evaluation.evaluator]: Inference done 2089/5000. 0.2080 s / img. ETA=0:10:15 [06/18 01:27:50 d2.evaluation.evaluator]: Inference done 2113/5000. 0.2080 s / img. ETA=0:10:10 [06/18 01:27:55 d2.evaluation.evaluator]: Inference done 2137/5000. 0.2079 s / img. ETA=0:10:05 [06/18 01:28:00 d2.evaluation.evaluator]: Inference done 2162/5000. 0.2079 s / img. ETA=0:10:00 [06/18 01:28:05 d2.evaluation.evaluator]: Inference done 2187/5000. 0.2078 s / img. ETA=0:09:54 [06/18 01:28:10 d2.evaluation.evaluator]: Inference done 2211/5000. 0.2078 s / img. ETA=0:09:49 [06/18 01:28:15 d2.evaluation.evaluator]: Inference done 2235/5000. 0.2079 s / img. ETA=0:09:44 [06/18 01:28:20 d2.evaluation.evaluator]: Inference done 2258/5000. 0.2080 s / img. ETA=0:09:40 [06/18 01:28:25 d2.evaluation.evaluator]: Inference done 2282/5000. 0.2080 s / img. ETA=0:09:35 [06/18 01:28:31 d2.evaluation.evaluator]: Inference done 2307/5000. 0.2079 s / img. ETA=0:09:29 [06/18 01:28:36 d2.evaluation.evaluator]: Inference done 2331/5000. 0.2079 s / img. ETA=0:09:24 [06/18 01:28:41 d2.evaluation.evaluator]: Inference done 2355/5000. 0.2080 s / img. ETA=0:09:19 [06/18 01:28:46 d2.evaluation.evaluator]: Inference done 2380/5000. 0.2079 s / img. ETA=0:09:14 [06/18 01:28:51 d2.evaluation.evaluator]: Inference done 2404/5000. 0.2079 s / img. ETA=0:09:09 [06/18 01:28:56 d2.evaluation.evaluator]: Inference done 2428/5000. 0.2080 s / img. ETA=0:09:04 [06/18 01:29:01 d2.evaluation.evaluator]: Inference done 2452/5000. 0.2080 s / img. ETA=0:08:59 [06/18 01:29:06 d2.evaluation.evaluator]: Inference done 2477/5000. 0.2079 s / img. ETA=0:08:53 [06/18 01:29:12 d2.evaluation.evaluator]: Inference done 2501/5000. 0.2079 s / img. ETA=0:08:48 [06/18 01:29:17 d2.evaluation.evaluator]: Inference done 2525/5000. 0.2079 s / img. ETA=0:08:43 [06/18 01:29:22 d2.evaluation.evaluator]: Inference done 2549/5000. 0.2079 s / img. ETA=0:08:38 [06/18 01:29:27 d2.evaluation.evaluator]: Inference done 2573/5000. 0.2079 s / img. ETA=0:08:33 [06/18 01:29:32 d2.evaluation.evaluator]: Inference done 2598/5000. 0.2078 s / img. ETA=0:08:27 [06/18 01:29:37 d2.evaluation.evaluator]: Inference done 2622/5000. 0.2079 s / img. ETA=0:08:22 [06/18 01:29:42 d2.evaluation.evaluator]: Inference done 2645/5000. 0.2079 s / img. ETA=0:08:18 [06/18 01:29:47 d2.evaluation.evaluator]: Inference done 2669/5000. 0.2079 s / img. ETA=0:08:13 [06/18 01:29:52 d2.evaluation.evaluator]: Inference done 2693/5000. 0.2079 s / img. ETA=0:08:08 [06/18 01:29:57 d2.evaluation.evaluator]: Inference done 2717/5000. 0.2080 s / img. ETA=0:08:03 [06/18 01:30:03 d2.evaluation.evaluator]: Inference done 2742/5000. 0.2079 s / img. ETA=0:07:57 [06/18 01:30:08 d2.evaluation.evaluator]: Inference done 2767/5000. 0.2079 s / img. ETA=0:07:52 [06/18 01:30:13 d2.evaluation.evaluator]: Inference done 2791/5000. 0.2079 s / img. ETA=0:07:47 [06/18 01:30:18 d2.evaluation.evaluator]: Inference done 2815/5000. 0.2079 s / img. ETA=0:07:42 [06/18 01:30:23 d2.evaluation.evaluator]: Inference done 2838/5000. 0.2079 s / img. ETA=0:07:37 [06/18 01:30:28 d2.evaluation.evaluator]: Inference done 2862/5000. 0.2079 s / img. ETA=0:07:32 [06/18 01:30:33 d2.evaluation.evaluator]: Inference done 2886/5000. 0.2080 s / img. ETA=0:07:27 [06/18 01:30:38 d2.evaluation.evaluator]: Inference done 2910/5000. 0.2080 s / img. ETA=0:07:22 [06/18 01:30:43 d2.evaluation.evaluator]: Inference done 2934/5000. 0.2080 s / img. ETA=0:07:17 [06/18 01:30:49 d2.evaluation.evaluator]: Inference done 2959/5000. 0.2080 s / img. ETA=0:07:11 [06/18 01:30:54 d2.evaluation.evaluator]: Inference done 2983/5000. 0.2080 s / img. ETA=0:07:06 [06/18 01:30:59 d2.evaluation.evaluator]: Inference done 3008/5000. 0.2080 s / img. ETA=0:07:01 [06/18 01:31:04 d2.evaluation.evaluator]: Inference done 3032/5000. 0.2080 s / img. ETA=0:06:56 [06/18 01:31:09 d2.evaluation.evaluator]: Inference done 3055/5000. 0.2081 s / img. ETA=0:06:51 [06/18 01:31:14 d2.evaluation.evaluator]: Inference done 3079/5000. 0.2081 s / img. ETA=0:06:46 [06/18 01:31:19 d2.evaluation.evaluator]: Inference done 3103/5000. 0.2081 s / img. ETA=0:06:41 [06/18 01:31:24 d2.evaluation.evaluator]: Inference done 3127/5000. 0.2081 s / img. ETA=0:06:36 [06/18 01:31:30 d2.evaluation.evaluator]: Inference done 3151/5000. 0.2081 s / img. ETA=0:06:31 [06/18 01:31:35 d2.evaluation.evaluator]: Inference done 3176/5000. 0.2081 s / img. ETA=0:06:26 [06/18 01:31:40 d2.evaluation.evaluator]: Inference done 3200/5000. 0.2081 s / img. ETA=0:06:20 [06/18 01:31:45 d2.evaluation.evaluator]: Inference done 3224/5000. 0.2080 s / img. ETA=0:06:15 [06/18 01:31:50 d2.evaluation.evaluator]: Inference done 3247/5000. 0.2081 s / img. ETA=0:06:11 [06/18 01:31:55 d2.evaluation.evaluator]: Inference done 3272/5000. 0.2081 s / img. ETA=0:06:05 [06/18 01:32:00 d2.evaluation.evaluator]: Inference done 3295/5000. 0.2081 s / img. ETA=0:06:00 [06/18 01:32:05 d2.evaluation.evaluator]: Inference done 3319/5000. 0.2081 s / img. ETA=0:05:55 [06/18 01:32:10 d2.evaluation.evaluator]: Inference done 3344/5000. 0.2081 s / img. ETA=0:05:50 [06/18 01:32:15 d2.evaluation.evaluator]: Inference done 3368/5000. 0.2081 s / img. ETA=0:05:45 [06/18 01:32:20 d2.evaluation.evaluator]: Inference done 3392/5000. 0.2081 s / img. ETA=0:05:40 [06/18 01:32:26 d2.evaluation.evaluator]: Inference done 3417/5000. 0.2080 s / img. ETA=0:05:34 [06/18 01:32:31 d2.evaluation.evaluator]: Inference done 3441/5000. 0.2080 s / img. ETA=0:05:29 [06/18 01:32:36 d2.evaluation.evaluator]: Inference done 3465/5000. 0.2081 s / img. ETA=0:05:24 [06/18 01:32:41 d2.evaluation.evaluator]: Inference done 3490/5000. 0.2080 s / img. ETA=0:05:19 [06/18 01:32:46 d2.evaluation.evaluator]: Inference done 3514/5000. 0.2080 s / img. ETA=0:05:14 [06/18 01:32:51 d2.evaluation.evaluator]: Inference done 3538/5000. 0.2080 s / img. ETA=0:05:09 [06/18 01:32:56 d2.evaluation.evaluator]: Inference done 3562/5000. 0.2080 s / img. ETA=0:05:04 [06/18 01:33:01 d2.evaluation.evaluator]: Inference done 3586/5000. 0.2080 s / img. ETA=0:04:59 [06/18 01:33:07 d2.evaluation.evaluator]: Inference done 3610/5000. 0.2081 s / img. ETA=0:04:54 [06/18 01:33:12 d2.evaluation.evaluator]: Inference done 3635/5000. 0.2080 s / img. ETA=0:04:48 [06/18 01:33:17 d2.evaluation.evaluator]: Inference done 3659/5000. 0.2081 s / img. ETA=0:04:43 [06/18 01:33:22 d2.evaluation.evaluator]: Inference done 3683/5000. 0.2081 s / img. ETA=0:04:38 [06/18 01:33:27 d2.evaluation.evaluator]: Inference done 3707/5000. 0.2081 s / img. ETA=0:04:33 [06/18 01:33:32 d2.evaluation.evaluator]: Inference done 3731/5000. 0.2081 s / img. ETA=0:04:28 [06/18 01:33:37 d2.evaluation.evaluator]: Inference done 3755/5000. 0.2081 s / img. ETA=0:04:23 [06/18 01:33:42 d2.evaluation.evaluator]: Inference done 3778/5000. 0.2081 s / img. ETA=0:04:18 [06/18 01:33:47 d2.evaluation.evaluator]: Inference done 3802/5000. 0.2081 s / img. ETA=0:04:13 [06/18 01:33:52 d2.evaluation.evaluator]: Inference done 3826/5000. 0.2081 s / img. ETA=0:04:08 [06/18 01:33:57 d2.evaluation.evaluator]: Inference done 3849/5000. 0.2082 s / img. ETA=0:04:03 [06/18 01:34:03 d2.evaluation.evaluator]: Inference done 3873/5000. 0.2082 s / img. ETA=0:03:58 [06/18 01:34:08 d2.evaluation.evaluator]: Inference done 3897/5000. 0.2082 s / img. ETA=0:03:53 [06/18 01:34:13 d2.evaluation.evaluator]: Inference done 3921/5000. 0.2082 s / img. ETA=0:03:48 [06/18 01:34:18 d2.evaluation.evaluator]: Inference done 3946/5000. 0.2081 s / img. ETA=0:03:43 [06/18 01:34:23 d2.evaluation.evaluator]: Inference done 3970/5000. 0.2081 s / img. ETA=0:03:38 [06/18 01:34:28 d2.evaluation.evaluator]: Inference done 3995/5000. 0.2081 s / img. ETA=0:03:32 [06/18 01:34:33 d2.evaluation.evaluator]: Inference done 4019/5000. 0.2081 s / img. ETA=0:03:27 [06/18 01:34:38 d2.evaluation.evaluator]: Inference done 4043/5000. 0.2081 s / img. ETA=0:03:22 [06/18 01:34:43 d2.evaluation.evaluator]: Inference done 4067/5000. 0.2081 s / img. ETA=0:03:17 [06/18 01:34:48 d2.evaluation.evaluator]: Inference done 4091/5000. 0.2081 s / img. ETA=0:03:12 [06/18 01:34:53 d2.evaluation.evaluator]: Inference done 4115/5000. 0.2081 s / img. ETA=0:03:07 [06/18 01:34:59 d2.evaluation.evaluator]: Inference done 4138/5000. 0.2081 s / img. ETA=0:03:02 [06/18 01:35:04 d2.evaluation.evaluator]: Inference done 4163/5000. 0.2081 s / img. ETA=0:02:57 [06/18 01:35:09 d2.evaluation.evaluator]: Inference done 4187/5000. 0.2081 s / img. ETA=0:02:52 [06/18 01:35:14 d2.evaluation.evaluator]: Inference done 4211/5000. 0.2081 s / img. ETA=0:02:47 [06/18 01:35:19 d2.evaluation.evaluator]: Inference done 4236/5000. 0.2081 s / img. ETA=0:02:41 [06/18 01:35:24 d2.evaluation.evaluator]: Inference done 4260/5000. 0.2081 s / img. ETA=0:02:36 [06/18 01:35:29 d2.evaluation.evaluator]: Inference done 4285/5000. 0.2081 s / img. ETA=0:02:31 [06/18 01:35:34 d2.evaluation.evaluator]: Inference done 4309/5000. 0.2081 s / img. ETA=0:02:26 [06/18 01:35:40 d2.evaluation.evaluator]: Inference done 4333/5000. 0.2081 s / img. ETA=0:02:21 [06/18 01:35:45 d2.evaluation.evaluator]: Inference done 4357/5000. 0.2081 s / img. ETA=0:02:16 [06/18 01:35:50 d2.evaluation.evaluator]: Inference done 4382/5000. 0.2081 s / img. ETA=0:02:10 [06/18 01:35:55 d2.evaluation.evaluator]: Inference done 4405/5000. 0.2081 s / img. ETA=0:02:05 [06/18 01:36:00 d2.evaluation.evaluator]: Inference done 4430/5000. 0.2081 s / img. ETA=0:02:00 [06/18 01:36:05 d2.evaluation.evaluator]: Inference done 4454/5000. 0.2081 s / img. ETA=0:01:55 [06/18 01:36:10 d2.evaluation.evaluator]: Inference done 4478/5000. 0.2081 s / img. ETA=0:01:50 [06/18 01:36:15 d2.evaluation.evaluator]: Inference done 4502/5000. 0.2081 s / img. ETA=0:01:45 [06/18 01:36:20 d2.evaluation.evaluator]: Inference done 4526/5000. 0.2081 s / img. ETA=0:01:40 [06/18 01:36:26 d2.evaluation.evaluator]: Inference done 4550/5000. 0.2081 s / img. ETA=0:01:35 [06/18 01:36:31 d2.evaluation.evaluator]: Inference done 4574/5000. 0.2081 s / img. ETA=0:01:30 [06/18 01:36:36 d2.evaluation.evaluator]: Inference done 4598/5000. 0.2081 s / img. ETA=0:01:25 [06/18 01:36:41 d2.evaluation.evaluator]: Inference done 4622/5000. 0.2081 s / img. ETA=0:01:20 [06/18 01:36:46 d2.evaluation.evaluator]: Inference done 4646/5000. 0.2081 s / img. ETA=0:01:14 [06/18 01:36:51 d2.evaluation.evaluator]: Inference done 4670/5000. 0.2081 s / img. ETA=0:01:09 [06/18 01:36:56 d2.evaluation.evaluator]: Inference done 4694/5000. 0.2081 s / img. ETA=0:01:04 [06/18 01:37:01 d2.evaluation.evaluator]: Inference done 4718/5000. 0.2081 s / img. ETA=0:00:59 [06/18 01:37:06 d2.evaluation.evaluator]: Inference done 4742/5000. 0.2081 s / img. ETA=0:00:54 [06/18 01:37:11 d2.evaluation.evaluator]: Inference done 4766/5000. 0.2082 s / img. ETA=0:00:49 [06/18 01:37:17 d2.evaluation.evaluator]: Inference done 4790/5000. 0.2082 s / img. ETA=0:00:44 [06/18 01:37:22 d2.evaluation.evaluator]: Inference done 4814/5000. 0.2082 s / img. ETA=0:00:39 [06/18 01:37:27 d2.evaluation.evaluator]: Inference done 4840/5000. 0.2081 s / img. ETA=0:00:33 [06/18 01:37:32 d2.evaluation.evaluator]: Inference done 4863/5000. 0.2082 s / img. ETA=0:00:29 [06/18 01:37:37 d2.evaluation.evaluator]: Inference done 4886/5000. 0.2082 s / img. ETA=0:00:24 [06/18 01:37:42 d2.evaluation.evaluator]: Inference done 4910/5000. 0.2082 s / img. ETA=0:00:19 [06/18 01:37:47 d2.evaluation.evaluator]: Inference done 4934/5000. 0.2082 s / img. ETA=0:00:13 [06/18 01:37:52 d2.evaluation.evaluator]: Inference done 4958/5000. 0.2082 s / img. ETA=0:00:08 [06/18 01:37:57 d2.evaluation.evaluator]: Inference done 4982/5000. 0.2082 s / img. ETA=0:00:03 [06/18 01:38:01 d2.evaluation.evaluator]: Total inference time: 0:17:37.639831 (0.211740 s / img per device, on 1 devices) [06/18 01:38:01 d2.evaluation.evaluator]: Total inference pure compute time: 0:17:19 (0.208200 s / img per device, on 1 devices) [06/18 01:38:01 d2.evaluation.coco_evaluation]: Preparing results for COCO format ... [06/18 01:38:01 d2.evaluation.coco_evaluation]: Saving results to ./output/coco_instances_results.json [06/18 01:38:01 d2.evaluation.coco_evaluation]: Evaluating predictions ... Loading and preparing results... DONE (t=0.01s) creating index... index created! Running per image evaluation... Evaluate annotation type bbox DONE (t=9.96s). Accumulating evaluation results... DONE (t=1.46s). Average Precision (AP) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.007 Average Precision (AP) @[ IoU=0.50 area= all maxDets=100 ] = 0.011 Average Precision (AP) @[ IoU=0.75 area= all maxDets=100 ] = 0.007 Average Precision (AP) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.005 Average Precision (AP) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.009 Average Precision (AP) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.009 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 1 ] = 0.003 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 10 ] = 0.009 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.009 Average Recall (AR) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.006 Average Recall (AR) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.011 Average Recall (AR) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.011 [06/18 01:38:13 d2.evaluation.coco_evaluation]: Evaluation results for bbox: AP AP50 AP75 APs APm APl
0.677 1.110 0.731 0.498 0.895 0.867
[06/18 01:38:13 d2.evaluation.coco_evaluation]: Per-category bbox AP: category AP category AP category AP
person 26.023 bicycle 0.003 car 27.923
motorcycle 0.000 airplane 0.011 bus 0.161
train 0.000 truck 0.000 boat 0.000
traffic light 0.000 fire hydrant 0.000 stop sign 0.000
parking meter 0.000 bench 0.000 bird 0.000
cat 0.000 dog 0.000 horse 0.000
sheep 0.000 cow 0.000 elephant 0.000
bear 0.000 zebra 0.000 giraffe 0.000
backpack 0.000 umbrella 0.000 handbag 0.000
tie 0.000 suitcase 0.000 frisbee 0.000
skis 0.000 snowboard 0.000 sports ball 0.000
kite 0.000 baseball bat 0.000 baseball glove 0.000
skateboard 0.000 surfboard 0.000 tennis racket 0.000
bottle 0.000 wine glass 0.000 cup 0.000
fork 0.000 knife 0.000 spoon 0.000
bowl 0.000 banana 0.000 apple 0.000
sandwich 0.000 orange 0.000 broccoli 0.000
carrot 0.000 hot dog 0.000 pizza 0.000
donut 0.000 cake 0.000 chair 0.000
couch 0.000 potted plant 0.000 bed 0.000
dining table 0.000 toilet 0.000 tv 0.000
laptop 0.000 mouse 0.000 remote 0.000
keyboard 0.000 cell phone 0.000 microwave 0.000
oven 0.000 toaster 0.000 sink 0.000
refrigerator 0.000 book 0.000 clock 0.000
vase 0.000 scissors 0.000 teddy bear 0.000
hair drier 0.000 toothbrush 0.000
Loading and preparing results... DONE (t=0.12s) creating index... index created! Running per image evaluation... Evaluate annotation type segm DONE (t=11.57s). Accumulating evaluation results... DONE (t=1.47s). Average Precision (AP) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.006 Average Precision (AP) @[ IoU=0.50 area= all maxDets=100 ] = 0.010 Average Precision (AP) @[ IoU=0.75 area= all maxDets=100 ] = 0.006 Average Precision (AP) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.004 Average Precision (AP) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.008 Average Precision (AP) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.008 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 1 ] = 0.003 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets= 10 ] = 0.008 Average Recall (AR) @[ IoU=0.50:0.95 area= all maxDets=100 ] = 0.008 Average Recall (AR) @[ IoU=0.50:0.95 area= small maxDets=100 ] = 0.006 Average Recall (AR) @[ IoU=0.50:0.95 area=medium maxDets=100 ] = 0.010 Average Recall (AR) @[ IoU=0.50:0.95 area= large maxDets=100 ] = 0.011 [06/18 01:38:26 d2.evaluation.coco_evaluation]: Evaluation results for segm: AP AP50 AP75 APs APm APl
0.578 1.047 0.580 0.383 0.797 0.847
[06/18 01:38:26 d2.evaluation.coco_evaluation]: Per-category segm AP: category AP category AP category AP
person 21.396 bicycle 0.000 car 24.650
motorcycle 0.000 airplane 0.019 bus 0.170
train 0.000 truck 0.000 boat 0.000
traffic light 0.000 fire hydrant 0.000 stop sign 0.000
parking meter 0.000 bench 0.000 bird 0.000
cat 0.000 dog 0.000 horse 0.000
sheep 0.000 cow 0.000 elephant 0.000
bear 0.000 zebra 0.000 giraffe 0.000
backpack 0.000 umbrella 0.000 handbag 0.000
tie 0.000 suitcase 0.000 frisbee 0.000
skis 0.000 snowboard 0.000 sports ball 0.000
kite 0.000 baseball bat 0.000 baseball glove 0.000
skateboard 0.000 surfboard 0.000 tennis racket 0.000
bottle 0.000 wine glass 0.000 cup 0.000
fork 0.000 knife 0.000 spoon 0.000
bowl 0.000 banana 0.000 apple 0.000
sandwich 0.000 orange 0.000 broccoli 0.000
carrot 0.000 hot dog 0.000 pizza 0.000
donut 0.000 cake 0.000 chair 0.000
couch 0.000 potted plant 0.000 bed 0.000
dining table 0.000 toilet 0.000 tv 0.000
laptop 0.000 mouse 0.000 remote 0.000
keyboard 0.000 cell phone 0.000 microwave 0.000
oven 0.000 toaster 0.000 sink 0.000
refrigerator 0.000 book 0.000 clock 0.000
vase 0.000 scissors 0.000 teddy bear 0.000
hair drier 0.000 toothbrush 0.000

OrderedDict([('bbox', {'AP': 0.6765157823530061, 'AP-airplane': 0.010954286918053509, 'AP-apple': 0.0, 'AP-backpack': 0.0, 'AP-banana': 0.0, 'AP-baseball bat': 0.0, 'AP-baseball glove': 0.0, 'AP-bear': 0.0, 'AP-bed': 0.0, 'AP-bench': 0.0, 'AP-bicycle': 0.0030003000300030005, 'AP-bird': 0.0, 'AP-boat': 0.0, 'AP-book': 0.0, 'AP-bottle': 0.0, 'AP-bowl': 0.0, 'AP-broccoli': 0.0, 'AP-bus': 0.1606446358921606, 'AP-cake': 0.0, 'AP-car': 27.92347658943828, 'AP-carrot': 0.0, 'AP-cat': 0.0, 'AP-cell phone': 0.0, 'AP-chair': 0.0, 'AP-clock': 0.0, 'AP-couch': 0.0, 'AP-cow': 0.0, 'AP-cup': 0.0, 'AP-dining table': 0.0, 'AP-dog': 0.0, 'AP-donut': 0.0, 'AP-elephant': 0.0, 'AP-fire hydrant': 0.0, 'AP-fork': 0.0, 'AP-frisbee': 0.0, 'AP-giraffe': 0.0, 'AP-hair drier': 0.0, 'AP-handbag': 0.0, 'AP-horse': 0.0, 'AP-hot dog': 0.0, 'AP-keyboard': 0.0, 'AP-kite': 0.0, 'AP-knife': 0.0, 'AP-laptop': 0.0, 'AP-microwave': 0.0, 'AP-motorcycle': 0.0, 'AP-mouse': 0.0, 'AP-orange': 0.0, 'AP-oven': 0.0, 'AP-parking meter': 0.0, 'AP-person': 26.023186775962, 'AP-pizza': 0.0, 'AP-potted plant': 0.0, 'AP-refrigerator': 0.0, 'AP-remote': 0.0, 'AP-sandwich': 0.0, 'AP-scissors': 0.0, 'AP-sheep': 0.0, 'AP-sink': 0.0, 'AP-skateboard': 0.0, 'AP-skis': 0.0, 'AP-snowboard': 0.0, 'AP-spoon': 0.0, 'AP-sports ball': 0.0, 'AP-stop sign': 0.0, 'AP-suitcase': 0.0, 'AP-surfboard': 0.0, 'AP-teddy bear': 0.0, 'AP-tennis racket': 0.0, 'AP-tie': 0.0, 'AP-toaster': 0.0, 'AP-toilet': 0.0, 'AP-toothbrush': 0.0, 'AP-traffic light': 0.0, 'AP-train': 0.0, 'AP-truck': 0.0, 'AP-tv': 0.0, 'AP-umbrella': 0.0, 'AP-vase': 0.0, 'AP-wine glass': 0.0, 'AP-zebra': 0.0, 'AP50': 1.1102748500813617, 'AP75': 0.7311378514151442, 'APl': 0.8669782881521491, 'APm': 0.8954265799842293, 'APs': 0.4977192938998091}), ('segm', {'AP': 0.5779413641930968, 'AP-airplane': 0.018959342742784917, 'AP-apple': 0.0, 'AP-backpack': 0.0, 'AP-banana': 0.0, 'AP-baseball bat': 0.0, 'AP-baseball glove': 0.0, 'AP-bear': 0.0, 'AP-bed': 0.0, 'AP-bench': 0.0, 'AP-bicycle': 0.0, 'AP-bird': 0.0, 'AP-boat': 0.0, 'AP-book': 0.0, 'AP-bottle': 0.0, 'AP-bowl': 0.0, 'AP-broccoli': 0.0, 'AP-bus': 0.17006248008110567, 'AP-cake': 0.0, 'AP-car': 24.65024903628289, 'AP-carrot': 0.0, 'AP-cat': 0.0, 'AP-cell phone': 0.0, 'AP-chair': 0.0, 'AP-clock': 0.0, 'AP-couch': 0.0, 'AP-cow': 0.0, 'AP-cup': 0.0, 'AP-dining table': 0.0, 'AP-dog': 0.0, 'AP-donut': 0.0, 'AP-elephant': 0.0, 'AP-fire hydrant': 0.0, 'AP-fork': 0.0, 'AP-frisbee': 0.0, 'AP-giraffe': 0.0, 'AP-hair drier': 0.0, 'AP-handbag': 0.0, 'AP-horse': 0.0, 'AP-hot dog': 0.0, 'AP-keyboard': 0.0, 'AP-kite': 0.0, 'AP-knife': 0.0, 'AP-laptop': 0.0, 'AP-microwave': 0.0, 'AP-motorcycle': 0.0, 'AP-mouse': 0.0, 'AP-orange': 0.0, 'AP-oven': 0.0, 'AP-parking meter': 0.0, 'AP-person': 21.39603827634095, 'AP-pizza': 0.0, 'AP-potted plant': 0.0, 'AP-refrigerator': 0.0, 'AP-remote': 0.0, 'AP-sandwich': 0.0, 'AP-scissors': 0.0, 'AP-sheep': 0.0, 'AP-sink': 0.0, 'AP-skateboard': 0.0, 'AP-skis': 0.0, 'AP-snowboard': 0.0, 'AP-spoon': 0.0, 'AP-sports ball': 0.0, 'AP-stop sign': 0.0, 'AP-suitcase': 0.0, 'AP-surfboard': 0.0, 'AP-teddy bear': 0.0, 'AP-tennis racket': 0.0, 'AP-tie': 0.0, 'AP-toaster': 0.0, 'AP-toilet': 0.0, 'AP-toothbrush': 0.0, 'AP-traffic light': 0.0, 'AP-train': 0.0, 'AP-truck': 0.0, 'AP-tv': 0.0, 'AP-umbrella': 0.0, 'AP-vase': 0.0, 'AP-wine glass': 0.0, 'AP-zebra': 0.0, 'AP50': 1.0466826769713014, 'AP75': 0.57953910118286, 'APl': 0.847422099235734, 'APm': 0.7967171769496791, 'APs': 0.383478473018425})])


## Expected behavior:

If there are no obvious error in "what you observed" provided above,
please tell us the expected behavior.

## Environment:

Provide your environment information using the following command:

Google colab



Thanks in advance!
Xuan Long
ppwwyyxx commented 4 years ago

This is working as expected because a model trained on cityscapes dataset is expected to not work on COCO dataset.

dxlong2000 commented 4 years ago

This is working as expected because a model trained on cityscapes dataset is expected to not work on COCO dataset.

Could you elaborate more why a model trained on cityscapes dataset is expected to not work on COCO dataset or could you give me more information about your point?

Thanks & BR