search

microsoft / X-Decoder

[CVPR 2023] Official Implementation of X-Decoder for generalized decoding for pixel, image and language
Apache License 2.0
1.29k stars 134 forks source link

PQ result when inference #20

Closed trqminh closed 1 year ago

trqminh commented 1 year ago

Hi, Thank you for your consideration. I am running this evaluation code with BestSeg Tiny checkpoint on ADE dataset, but I cannot see where is PQ showing in the output. Could you show me how I can get the value? 

mpirun -n 8 python eval.py evaluate --conf_files configs/xdecoder/svlp_focalt_lang.yaml  --overrides WEIGHT [BestSeg Tiny](https://projects4jw.blob.core.windows.net/x-decoder/release/xdecoder_focalt_best_openseg.pt)
INFO:datasets.evaluation.segmentation_evaluation:OrderedDict([('sem_seg', {'mIoU': 24.906070100032558, 'fwIoU': 56.48813787121508, 'IoU-wall': 66.61977529364381, 'IoU-building': 61.686172646915516
, 'IoU-sky': 93.01439625679751, 'IoU-floor': 55.50932632362232, 'IoU-tree': 64.54442464927908, 'IoU-ceiling': 77.49496143151943, 'IoU-road, route': 77.31548965512836, 'IoU-bed': 77.4902292301069, 
'IoU-window ': 48.1756515295317, 'IoU-grass': 60.422415485190086, 'IoU-cabinet': 48.013506766288764, 'IoU-sidewalk, pavement': 55.61478885979676, 'IoU-person': 82.24137142446519, 'IoU-earth, groun
d': 0.0, 'IoU-door': 36.11607509482755, 'IoU-table': 27.384548258439782, 'IoU-mountain, mount': 40.955413211111924, 'IoU-plant': 18.663105611819013, 'IoU-curtain': 64.62422235351946, 'IoU-chair': 
39.0646160710763, 'IoU-car': 75.91038694971013, 'IoU-water': 21.279867350426827, 'IoU-painting, picture': 9.535676742284622, 'IoU-sofa': 57.18693982433452, 'IoU-shelf': 29.684473915323572, 'IoU-ho
use': 19.03444434625438, 'IoU-sea': 43.15172790035995, 'IoU-mirror': 60.23490728806231, 'IoU-rug': 21.813911723878732, 'IoU-field': 20.382156190607574, 'IoU-armchair': 24.99894621869992, 'IoU-seat
': 18.71646580465292, 'IoU-fence': 26.003060872301774, 'IoU-desk': 7.8400777016298075, 'IoU-rock, stone': 25.894778455588952, 'IoU-wardrobe, closet, press': 0.1436282418813819, 'IoU-lamp': 0.54134
76360545296, 'IoU-tub': 54.26845967428996, 'IoU-rail': 8.755158777335478, 'IoU-cushion': 0.41486664629697106, 'IoU-base, pedestal, stand': 0.0, 'IoU-box': 14.633044429384977, 'IoU-column, pillar':
 0.01473894282814901, 'IoU-signboard, sign': 17.25362379657421, 'IoU-chest of drawers, chest, bureau, dresser': 3.856919525909023, 'IoU-counter': 3.5828783721346875, 'IoU-sand': 42.102149254481276
, 'IoU-sink': 57.40884944322997, 'IoU-skyscraper': 0.00863472303898623, 'IoU-fireplace': 22.032838709976183, 'IoU-refrigerator, icebox': 56.98516552782412, 'IoU-grandstand, covered stand': 25.7227
0214487473, 'IoU-path': 7.399178947435783, 'IoU-stairs': 21.38567914570656, 'IoU-runway': 33.59013043788163, 'IoU-case, display case, showcase, vitrine': 0.0, 'IoU-pool table, billiard table, snoo
ker table': 74.53544749221741, 'IoU-pillow': 0.07518131113896317, 'IoU-screen door, screen': 0.0, 'IoU-stairway, staircase': 0.43446658348730305, 'IoU-river': 10.924773698806375, 'IoU-bridge, span
': 28.482600059873786, 'IoU-bookcase': 0.5151970123771235, 'IoU-blind, screen': 27.547533785121125, 'IoU-coffee table': 0.8817475730284592, 'IoU-toilet, can, commode, crapper, pot, potty, stool, t
hrone': 82.89734111449071, 'IoU-flower': 28.097456972058275, 'IoU-book': 36.279829668107304, 'IoU-hill': 0.041143133735352846, 'IoU-bench': 33.56947155364012, 'IoU-countertop': 16.015611690381427,
 'IoU-stove': 10.272521332419522, 'IoU-palm, palm tree': 2.3869788714500717, 'IoU-kitchen island': 21.81746243524508, 'IoU-computer': 17.14787349332021, 'IoU-swivel chair': 20.157961725297216, 'Io
U-boat': 68.79855577096204, 'IoU-bar': 0.02561358846374377, 'IoU-arcade machine': 10.70786438145052, 'IoU-hovel, hut, hutch, shack, shanty': 3.0488305908362476, 'IoU-bus': 70.62583672672505, 'IoU-
towel': 52.92953584348092, 'IoU-light': 11.821621424549436, 'IoU-truck': 17.873127725791846, 'IoU-tower': 4.224330548274466, 'IoU-chandelier': 1.1220895092091254, 'IoU-awning, sunshade, sunblind':
 6.616024058269303, 'IoU-street lamp': 0.0, 'IoU-booth': 0.15086258505263087, 'IoU-tv': 23.65220188276453, 'IoU-plane': 56.773936346924756, 'IoU-dirt track': 0.43521387670587325, 'IoU-clothes': 21
.258650644694562, 'IoU-pole': 19.32663959381968, 'IoU-land, ground, soil': 0.46559736720594347, 'IoU-bannister, banister, balustrade, balusters, handrail': 0.560122796151464, 'IoU-escalator, movin
g staircase, moving stairway': 0.0162891978031097, 'IoU-ottoman, pouf, pouffe, puff, hassock': 23.25492985064977, 'IoU-bottle': 28.12634833884654, 'IoU-buffet, counter, sideboard': 0.2758246120797
785, 'IoU-poster, posting, placard, notice, bill, card': 19.13703697778062, 'IoU-stage': 1.77526462298315, 'IoU-van': 25.709683301926933, 'IoU-ship': 7.068624870371017, 'IoU-fountain': 3.101805718
290426, 'IoU-conveyer belt, conveyor belt, conveyer, conveyor, transporter': 16.533787213711964, 'IoU-canopy': 0.0, 'IoU-washer, automatic washer, washing machine': 22.37631128950778, 'IoU-playthi
ng, toy': 5.314420570382705, 'IoU-pool': 20.959738781790747, 'IoU-stool': 16.827886868101345, 'IoU-barrel, cask': 4.834328636287863, 'IoU-basket, handbasket': 11.583300088020923, 'IoU-falls': 32.7
7802080487416, 'IoU-tent': 71.40640707727364, 'IoU-bag': 13.473183014587715, 'IoU-minibike, motorbike': 63.639934684823466, 'IoU-cradle': 10.330452727976267, 'IoU-oven': 12.40630869557509, 'IoU-ba
ll': 27.72555385976339, 'IoU-food, solid food': 44.03799235138715, 'IoU-step, stair': 0.11994717922382803, 'IoU-tank, storage tank': 14.484766361076412, 'IoU-trade name': 1.4821179250349048, 'IoU-
microwave': 81.64071768382925, 'IoU-pot': 11.439637772080083, 'IoU-animal': 65.39664495514317, 'IoU-bicycle': 55.0969895488581, 'IoU-lake': 0.0035481019133139567, 'IoU-dishwasher': 2.4782123291820
644, 'IoU-screen': 0.0, 'IoU-blanket, cover': 0.09492631909517851, 'IoU-sculpture': 14.610162806171632, 'IoU-hood, exhaust hood': 0.0, 'IoU-sconce': 0.3579455589777741, 'IoU-vase': 21.379527228890
403, 'IoU-traffic light': 23.907536415452817, 'IoU-tray': 5.549347607095087, 'IoU-trash can': 21.94514409763293, 'IoU-fan': 6.584765209604298, 'IoU-pier': 52.210612766345754, 'IoU-crt screen': 7.6
8999518999519, 'IoU-plate': 14.452253891471493, 'IoU-monitor': 19.805141076205974, 'IoU-bulletin board': 15.188605496920054, 'IoU-shower': 1.1203642318951137, 'IoU-radiator': 22.429703078072507, '
IoU-glass, drinking glass': 22.770563080904573, 'IoU-clock': 19.53432728011467, 'IoU-flag': 42.22801707703967, 'mACC': 40.35234468002594, 'pACC': 68.7597599281644, 'ACC-wall': 75.15670120826825, '
ACC-building': 68.67683393707412, 'ACC-sky': 95.78909199997325, 'ACC-floor': 60.22084554380375, 'ACC-tree': 92.80746256443825, 'ACC-ceiling': 89.93663735376995, 'ACC-road, route': 87.8153535576437
9, 'ACC-bed': 94.38398201483389, 'ACC-window ': 70.65793138930107, 'ACC-grass': 84.02449503110103, 'ACC-cabinet': 75.40193167798917, 'ACC-sidewalk, pavement': 74.67516786509663, 'ACC-person': 93.4
3694710780233, 'ACC-earth, ground': 0.0, 'ACC-door': 58.60643747791655, 'ACC-table': 48.870157085166504, 'ACC-mountain, mount': 54.60010698573554, 'ACC-plant': 21.0733259943901, 'ACC-curtain': 87.
95054658274954, 'ACC-chair': 50.591434334198006, 'ACC-car': 84.48022360668509, 'ACC-water': 27.84542279761975, 'ACC-painting, picture': 9.731982703479733, 'ACC-sofa': 83.30536364230761, 'ACC-shelf
': 62.63653966371074, 'ACC-house': 81.32707087266269, 'ACC-sea': 65.19502955657389, 'ACC-mirror': 78.17358618008736, 'ACC-rug': 88.76685399337744, 'ACC-field': 40.696410424907555, 'ACC-armchair': 
58.069813501995185, 'ACC-seat': 24.152624100075183, 'ACC-fence': 68.61097175507057, 'ACC-desk': 10.22930868729243, 'ACC-rock, stone': 78.59270114544158, 'ACC-wardrobe, closet, press': 0.1457117503
9727374, 'ACC-lamp': 0.5445746343245543, 'ACC-tub': 57.88049209449019, 'ACC-rail': 20.92687386168558, 'ACC-cushion': 0.4199210621996574, 'ACC-base, pedestal, stand': 0.0, 'ACC-box': 16.91968602077
615, 'ACC-column, pillar': 0.014741265376664746, 'ACC-signboard, sign': 18.173210119568388, 'ACC-chest of drawers, chest, bureau, dresser': 6.056264461843328, 'ACC-counter': 5.369140951577996, 'AC
C-sand': 79.06253722594334, 'ACC-sink': 77.63702513641833, 'ACC-skyscraper': 0.008999024316310968, 'ACC-fireplace': 33.242078468315626, 'ACC-refrigerator, icebox': 80.61806003729038, 'ACC-grandsta
nd, covered stand': 55.012662215203775, 'ACC-path': 8.455863083593265, 'ACC-stairs': 34.94661706879935, 'ACC-runway': 45.469493363822686, 'ACC-case, display case, showcase, vitrine': 0.0, 'ACC-poo
l table, billiard table, snooker table': 79.12364012022634, 'ACC-pillow': 0.08182200100365528, 'ACC-screen door, screen': 0.0, 'ACC-stairway, staircase': 0.45854961938294114, 'ACC-river': 53.40964
6868303604, 'ACC-bridge, span': 91.17194986492365, 'ACC-bookcase': 0.5256641175594662, 'ACC-blind, screen': 44.68027965107038, 'ACC-coffee table': 1.1000662146558753, 'ACC-toilet, can, commode, cr
apper, pot, potty, stool, throne': 89.06724591073926, 'ACC-flower': 58.852205974377355, 'ACC-book': 57.24606762924744, 'ACC-hill': 0.041354229382179974, 'ACC-bench': 67.33395668697592, 'ACC-counte
rtop': 54.03231228704103, 'ACC-stove': 11.937893768857206, 'ACC-palm, palm tree': 2.63794537266041, 'ACC-kitchen island': 27.909905774199434, 'ACC-computer': 18.984995644741925, 'ACC-swivel chair'
: 49.960279406736305, 'ACC-boat': 84.16026197565539, 'ACC-bar': 0.031770913970170823, 'ACC-arcade machine': 14.247488261725652, 'ACC-hovel, hut, hutch, shack, shanty': 14.606730445461208, 'ACC-bus
': 95.69015807634807, 'ACC-towel': 70.05251673009853, 'ACC-light': 67.63898063484336, 'ACC-truck': 77.54445385266723, 'ACC-tower': 5.937523893669983, 'ACC-chandelier': 1.1684525706219897, 'ACC-awn
ing, sunshade, sunblind': 6.683245602112896, 'ACC-street lamp': 0.0, 'ACC-booth': 0.31014948361165906, 'ACC-tv': 88.33659716622509, 'ACC-plane': 65.70826800415259, 'ACC-dirt track': 20.84110890936
7247, 'ACC-clothes': 44.47192964329344, 'ACC-pole': 24.358429311277728, 'ACC-land, ground, soil': 2.5005435964340075, 'ACC-bannister, banister, balustrade, balusters, handrail': 1.188901409370764,
 'ACC-escalator, moving staircase, moving stairway': 0.01704667895314093, 'ACC-ottoman, pouf, pouffe, puff, hassock': 32.68594650789151, 'ACC-bottle': 49.44455857023213, 'ACC-buffet, counter, side
board': 0.575008821879276, 'ACC-poster, posting, placard, notice, bill, card': 43.11282519543765, 'ACC-stage': 6.299271112652419, 'ACC-van': 28.385769962751855, 'ACC-ship': 7.534140075716604, 'ACC
-fountain': 3.7697330071891897, 'ACC-conveyer belt, conveyor belt, conveyer, conveyor, transporter': 57.181953140127796, 'ACC-canopy': 0.0, 'ACC-washer, automatic washer, washing machine': 22.4823
30654615268, 'ACC-plaything, toy': 20.16468316860214, 'ACC-pool': 39.71261974177426, 'ACC-stool': 40.52768837010897, 'ACC-barrel, cask': 56.09284332688588, 'ACC-basket, handbasket': 20.20389454035
3666, 'ACC-falls': 97.21638040459206, 'ACC-tent': 97.23345227556878, 'ACC-bag': 20.450235714746995, 'ACC-minibike, motorbike': 90.5424878500474, 'ACC-cradle': 30.83022818957288, 'ACC-oven': 75.076
36381995809, 'ACC-ball': 34.179809213364656, 'ACC-food, solid food': 91.2132115723329, 'ACC-step, stair': 0.12409349135330214, 'ACC-tank, storage tank': 47.44675391326661, 'ACC-trade name': 1.5337
997519713598, 'ACC-microwave': 94.3721055965209, 'ACC-pot': 15.866938656916988, 'ACC-animal': 86.82721077890477, 'ACC-bicycle': 84.62376841921701, 'ACC-lake': 0.003810539953511413, 'ACC-dishwasher
': 3.5840118152860265, 'ACC-screen': 0.0, 'ACC-blanket, cover': 0.13857919853293602, 'ACC-sculpture': 64.88891435681658, 'ACC-hood, exhaust hood': 0.0, 'ACC-sconce': 0.4362750900451494, 'ACC-vase : 71.10418893006829, 'ACC-traffic light': 32.50758035172832, 'ACC-tray': 16.195304534852863, 'ACC-trash can': 26.094582519773635, 'ACC-fan': 9.99769344696142, 'ACC-pier': 79.61677079999443, 'ACC-crt screen': 17.499422785849276, 'ACC-plate': 25.26266284611618, 'ACC-monitor': 25.13872113987642, 'ACC-bulletin board': 47.330646015438575, 'ACC-shower': 17.920782490799922, 'ACC-radiator': 23.469
234552561414, 'ACC-glass, drinking glass': 28.009329682587975, 'ACC-clock': 32.82381945761426, 'ACC-flag': 53.67568311405477})])
INFO:detectron2.evaluation.coco_evaluation:Preparing results for COCO format ...                                                                                                                    
INFO:detectron2.evaluation.coco_evaluation:Saving results to ../data/train_outputs/xdecoder/test/coco_instances_results.json                                                                        
INFO:detectron2.evaluation.coco_evaluation:Evaluating predictions with unofficial COCO API...                                                                                                       
Loading and preparing results...                                                                  
DONE (t=0.07s)                                                                                    
creating index...                                                                                 
index created!                                                                                    
INFO:detectron2.evaluation.fast_eval_api:Evaluate annotation type *bbox*                                                                                                                            
INFO:detectron2.evaluation.fast_eval_api:COCOeval_opt.evaluate() finished in 4.56 seconds.                                                                                                          
INFO:detectron2.evaluation.fast_eval_api:Accumulating evaluation results...                                                                                                                         
INFO:detectron2.evaluation.fast_eval_api:COCOeval_opt.accumulate() finished in 0.65 seconds.                                                                                                        
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000                                                                                                                     
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000                                                                                                                     
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000                                                                                                                     
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000                                                                                                                     
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000                                                                                                                     
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000                                                                                                                     
INFO:detectron2.evaluation.coco_evaluation:Evaluation results for bbox:                                                                                                                             
|  AP   |  AP50  |  AP75  |  APs  |  APm  |  APl  |                                                                                                                                                 
|:-----:|:------:|:------:|:-----:|:-----:|:-----:|                                                                                                                                                 
| 0.000 | 0.000  | 0.000  | 0.000 | 0.000 | 0.000 |                                                                                                                                                 
INFO:detectron2.evaluation.coco_evaluation:Per-category bbox AP:                                                                                                                                    
| category                   | AP    | category                                                 | AP    | category                                  | AP    |
|:---------------------------|:------|:---------------------------------------------------------|:------|:------------------------------------------|:------|
| bed                        | 0.000 | window                                                   | 0.000 | cabinet                                   | 0.000 |
| person                     | 0.000 | door                                                     | 0.000 | table                                     | 0.000 |
| curtain                    | 0.000 | chair                                                    | 0.000 | car                                       | 0.000 |
| painting, picture          | 0.000 | sofa                                                     | 0.000 | shelf                                     | 0.000 |
| mirror                     | 0.000 | armchair                                                 | 0.000 | seat                                      | 0.000 |
| fence                      | 0.000 | desk                                                     | 0.000 | wardrobe, closet, press                   | 0.000 |
| lamp                       | 0.000 | tub                                                      | 0.000 | rail                                      | 0.000 |
| cushion                    | 0.000 | box                                                      | 0.000 | column, pillar                            | 0.000 |
| signboard, sign            | 0.000 | chest of drawers, chest, bureau, dresser                 | 0.000 | counter                                   | 0.000 |
| sink                       | 0.000 | fireplace                                                | 0.000 | refrigerator, icebox                      | 0.000 |
| stairs                     | 0.000 | case, display case, showcase, vitrine                    | 0.000 | pool table, billiard table, snooker table | 0.000 |
| pillow                     | 0.000 | screen door, screen                                      | 0.000 | bookcase                                  | 0.000 |
| coffee table               | 0.000 | toilet, can, commode, crapper, pot, potty, stool, throne | 0.000 | flower                                    | 0.000 |
| book                       | 0.000 | bench                                                    | 0.000 | countertop                                | 0.000 |
| stove                      | 0.000 | palm, palm tree                                          | 0.000 | kitchen island                            | 0.000 |
| computer                   | 0.000 | swivel chair                                             | 0.000 | boat                                      | 0.000 |
| arcade machine             | 0.000 | bus                                                      | 0.000 | towel                                     | 0.000 |
| light                      | 0.000 | truck                                                    | 0.000 | chandelier                                | 0.000 |
| awning, sunshade, sunblind | 0.000 | street lamp                                              | 0.000 | booth                                     | 0.000 |
| tv                         | 0.000 | plane                                                    | 0.000 | clothes                                   | 0.000 |
| pole                       | 0.000 | bannister, banister, balustrade, balusters, handrail     | 0.000 | ottoman, pouf, pouffe, puff, hassock      | 0.000 |
| bottle                     | 0.000 | van                                                      | 0.000 | ship                                      | 0.000 |
| fountain                   | 0.000 | washer, automatic washer, washing machine                | 0.000 | plaything, toy                            | 0.000 |
| stool                      | 0.000 | barrel, cask                                             | 0.000 | basket, handbasket                        | 0.000 |
| bag                        | 0.000 | minibike, motorbike                                      | 0.000 | oven                                      | 0.000 |
| ball                       | 0.000 | food, solid food                                         | 0.000 | step, stair                               | 0.000 |
| trade name                 | 0.000 | microwave                                                | 0.000 | pot                                       | 0.000 |                                       
| animal                     | 0.000 | bicycle                                                  | 0.000 | dishwasher                                | 0.000 |                                       
| screen                     | 0.000 | sculpture                                                | 0.000 | hood, exhaust hood                        | 0.000 |                                       
| sconce                     | 0.000 | vase                                                     | 0.000 | traffic light                             | 0.000 |                                       
| tray                       | 0.000 | trash can                                                | 0.000 | fan                                       | 0.000 |                                       
| plate                      | 0.000 | monitor                                                  | 0.000 | bulletin board                            | 0.000 |                                       
| radiator                   | 0.000 | glass, drinking glass                                    | 0.000 | clock                                     | 0.000 |                                       
| flag                       | 0.000 |                                                          |       |                                           |       |                                       
Loading and preparing results...                                                                                                                                                                    
DONE (t=0.87s)                                                                                                                                                                                      
creating index...                                                                                                                                                                                   
index created!                                                                                                                                                                                      
INFO:detectron2.evaluation.fast_eval_api:Evaluate annotation type *segm*                                                                                                                            
INFO:detectron2.evaluation.fast_eval_api:COCOeval_opt.evaluate() finished in 4.98 seconds.                                                                                                          
INFO:detectron2.evaluation.fast_eval_api:Accumulating evaluation results...                                                                                                                         
INFO:detectron2.evaluation.fast_eval_api:COCOeval_opt.accumulate() finished in 0.60 seconds.                                                                                                        
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.100                                                                                                                     
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.180                                                                                                                     
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.095                                                                                                                     
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.034                                                                                                                     
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.118                                                                                                                     
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.210                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.153                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.238                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.246                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.092                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.265                                                                                                                     
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.399                                                                                                                     
INFO:detectron2.evaluation.coco_evaluation:Evaluation results for segm:                                                                                                                             
|  AP   |  AP50  |  AP75  |  APs  |  APm   |  APl   |                                                                                                                                               
|:-----:|:------:|:------:|:-----:|:------:|:------:|                                                                                                                                               
| 9.953 | 17.999 | 9.532  | 3.372 | 11.822 | 20.981 |                                                                                                                                               
INFO:detectron2.evaluation.coco_evaluation:Per-category segm AP:                                                                                                                                    
| category                   | AP     | category                                                 | AP     | category                                  | AP     |
|:---------------------------|:-------|:---------------------------------------------------------|:-------|:------------------------------------------|:-------|
| bed                        | 39.526 | window                                                   | 14.031 | cabinet                                   | 4.851  |
| person                     | 25.881 | door                                                     | 13.307 | table                                     | 3.983  |
| curtain                    | 19.278 | chair                                                    | 13.749 | car                                       | 30.366 |
| painting, picture          | 1.841  | sofa                                                     | 35.950 | shelf                                     | 2.820  |
| mirror                     | 35.698 | armchair                                                 | 16.265 | seat                                      | 1.160  |
| fence                      | 2.201  | desk                                                     | 1.411  | wardrobe, closet, press                   | 0.025  |
| lamp                       | 3.738  | tub                                                      | 19.004 | rail                                      | 0.470  |
| cushion                    | 3.202  | box                                                      | 2.820  | column, pillar                            | 1.295  |
| signboard, sign            | 5.334  | chest of drawers, chest, bureau, dresser                 | 9.151  | counter                                   | 0.226  |
| sink                       | 21.508 | fireplace                                                | 4.796  | refrigerator, icebox                      | 54.810 |
| stairs                     | 6.297  | case, display case, showcase, vitrine                    | 0.019  | pool table, billiard table, snooker table | 45.733 |
| pillow                     | 0.389  | screen door, screen                                      | 0.183  | bookcase                                  | 0.071  |
| coffee table               | 12.674 | toilet, can, commode, crapper, pot, potty, stool, throne | 51.184 | flower                                    | 6.356  |
| book                       | 1.527  | bench                                                    | 3.421  | countertop                                | 3.035  |
| stove                      | 13.380 | palm, palm tree                                          | 1.796  | kitchen island                            | 18.354 |
| computer                   | 0.644  | swivel chair                                             | 6.743  | boat                                      | 12.381 |
| arcade machine             | 8.560  | bus                                                      | 36.174 | towel                                     | 13.658 |
| light                      | 0.660  | truck                                                    | 9.730  | chandelier                                | 2.660  |
| awning, sunshade, sunblind | 4.058  | street lamp                                              | 0.047  | booth                                     | 0.246  |
| tv                         | 50.874 | plane                                                    | 20.882 | clothes                                   | 0.990  |
| pole                       | 0.214  | bannister, banister, balustrade, balusters, handrail     | 0.093  | ottoman, pouf, pouffe, puff, hassock      | 9.956  |
| bottle                     | 9.767  | van                                                      | 11.846 | ship                                      | 13.218 |
| fountain                   | 0.703  | washer, automatic washer, washing machine                | 4.326  | plaything, toy                            | 0.099  |
| stool                      | 3.533  | barrel, cask                                             | 3.285  | basket, handbasket                        | 3.355  |
| bag                        | 2.679  | minibike, motorbike                                      | 18.870 | oven                                      | 9.317  |
| ball                       | 7.168  | food, solid food                                         | 0.943  | step, stair                               | 0.691  |
| trade name                 | 0.839  | microwave                                                | 56.907 | pot                                       | 0.579  |
| animal                     | 14.792 | bicycle                                                  | 7.610  | dishwasher                                | 2.745  |
| screen                     | 0.264  | sculpture                                                | 5.385  | hood, exhaust hood                        | 0.000  |
| sconce                     | 0.851  | vase                                                     | 15.081 | traffic light                             | 4.156  |
| tray                       | 0.703  | trash can                                                | 8.347  | fan                                       | 0.497  |
| plate                      | 0.300  | monitor                                                  | 5.606  | bulletin board                            | 0.783  |
| radiator                   | 14.912 | glass, drinking glass                                    | 7.721  | clock                                     | 17.529 |
| flag                       | 8.235  |                                                          |        |                                           |        |
INFO:__main__:{'ade20k_panoptic_val/sem_seg/mIoU': 24.906070100032558, 'ade20k_panoptic_val/bbox/AP': 0.0, 'ade20k_panoptic_val/segm/AP': 9.953305772030166}
MaureenZOU commented 1 year ago

Thanks so much for your interests in our work! I just pull the code and evaluate ade segmentation, the PQ directly comes after mIoU results as shown below:

Screenshot 2023-03-02 at 4 21 41 PM

Could you please check whether you have open the switcher of PQ shown below? Thanks!

Screenshot 2023-03-02 at 4 21 50 PM
trqminh commented 1 year ago

my bad, I turn the panoptic_on to false for debugging but forgot to turn it back on. Thank you for your quick response. Btw, just a one quick question, if I just test on this dataset, as attached, what is the annotation json file name that the code use for evaluation. (Since there is a lot of json file that I downloaded and prepared to run the code) image

MaureenZOU commented 1 year ago

Please refer to the files below, the data registration happens here:

Screenshot 2023-03-02 at 4 45 12 PM
trqminh commented 1 year ago

Thank you.