BEIT segmentation evalutaion

[yetash]

I am using BEIT segmentation and try to test it on my own. I run the following codes on instruction:

bash tools/dist_test.sh configs/beit/upernet/upernet_beit_base_12_640_slide_160k_ade20k_pt2ft.py https://unilm.blob.core.windows.net/beit/beit_base_patch16_640_pt22k_ft22ktoade20k.pth 1 --eval mIoU

but I get results lower than expected:

+---------------------+-------+-------+ | Class | IoU | Acc | +---------------------+-------+-------+ | wall | 43.84 | 92.17 | | building | 62.88 | 72.84 | | sky | 74.12 | 76.54 | | floor | 57.22 | 77.03 | | tree | 59.4 | 71.83 | | ceiling | 60.99 | 64.1 | | road | 62.43 | 73.12 | | bed | 65.84 | 69.75 | | windowpane | 49.64 | 63.13 | | grass | 60.02 | 75.71 | | cabinet | 39.64 | 55.88 | | sidewalk | 48.01 | 54.85 | | person | 65.56 | 74.07 | | earth | 27.82 | 36.43 | | door | 30.14 | 37.47 | | table | 42.17 | 55.6 | | mountain | 39.54 | 43.71 | | plant | 44.92 | 53.06 | | curtain | 55.07 | 61.78 | | chair | 43.97 | 60.39 | | car | 63.23 | 68.7 | | water | 52.73 | 66.54 | | painting | 46.53 | 51.96 | | sofa | 54.49 | 64.29 | | shelf | 26.78 | 31.63 | | house | 6.81 | 7.32 | | sea | 46.16 | 60.29 | | mirror | 27.6 | 28.59 | | rug | 33.17 | 34.61 | | field | 15.94 | 19.99 | | armchair | 27.86 | 36.36 | | seat | 53.05 | 56.72 | | fence | 21.7 | 24.75 | | desk | 14.2 | 15.58 | | rock | 49.11 | 65.44 | | wardrobe | 14.63 | 17.31 | | lamp | 44.04 | 49.18 | | bathtub | 38.21 | 39.05 | | railing | 19.17 | 23.11 | | cushion | 51.41 | 69.84 | | base | 9.48 | 10.39 | | box | 18.91 | 29.75 | | column | 32.48 | 37.78 | | signboard | 28.38 | 44.63 | | chest of drawers | 20.6 | 27.7 | | counter | 3.01 | 3.41 | | sand | 14.36 | 18.3 | | sink | 44.72 | 49.9 | | skyscraper | 39.11 | 40.85 | | fireplace | 25.12 | 27.08 | | refrigerator | 28.36 | 29.13 | | grandstand | 10.28 | 10.76 | | path | 22.11 | 26.76 | | stairs | 27.0 | 32.18 | | runway | 34.65 | 39.88 | | case | 26.87 | 28.9 | | pool table | 29.7 | 30.1 | | pillow | 41.51 | 45.57 | | screen door | 17.1 | 17.33 | | stairway | 13.16 | 15.21 | | river | 6.94 | 9.38 | | bridge | 26.51 | 26.95 | | bookcase | 8.3 | 10.08 | | blind | 33.3 | 34.8 | | coffee table | 34.55 | 40.76 | | toilet | 40.22 | 41.85 | | flower | 31.91 | 42.75 | | book | 37.71 | 52.66 | | hill | 1.51 | 1.78 | | bench | 16.72 | 18.53 | | countertop | 23.03 | 27.18 | | stove | 33.45 | 35.55 | | palm | 23.05 | 27.53 | | kitchen island | 21.32 | 29.07 | | computer | 10.73 | 10.93 | | swivel chair | 10.82 | 12.03 | | boat | 13.9 | 14.29 | | bar | 7.87 | 8.35 | | arcade machine | 33.15 | 34.41 | | hovel | 20.56 | 20.82 | | bus | 42.27 | 43.31 | | towel | 49.91 | 65.05 | | light | 15.24 | 49.89 | | truck | 31.86 | 38.41 | | tower | 2.06 | 2.12 | | chandelier | 29.64 | 31.77 | | awning | 8.44 | 9.32 | | streetlight | 21.27 | 26.0 | | booth | 38.83 | 39.27 | | television receiver | 26.22 | 29.63 | | airplane | 51.4 | 53.54 | | dirt track | 14.82 | 15.18 | | apparel | 19.99 | 23.0 | | pole | 21.52 | 29.16 | | land | 4.11 | 5.78 | | bannister | 2.93 | 3.65 | | escalator | 27.21 | 31.6 | | ottoman | 18.44 | 21.1 | | bottle | 32.37 | 51.3 | | buffet | 46.49 | 48.87 | | poster | 16.63 | 22.79 | | stage | 2.45 | 2.91 | | van | 23.87 | 28.62 | | ship | 0.0 | 0.0 | | fountain | 0.06 | 0.07 | | conveyer belt | 31.86 | 33.62 | | canopy | 2.96 | 3.31 | | washer | 6.12 | 6.19 | | plaything | 10.21 | 13.06 | | swimming pool | 35.98 | 40.05 | | stool | 34.88 | 44.15 | | barrel | 4.99 | 5.2 | | basket | 24.38 | 28.33 | | waterfall | 33.87 | 36.1 | | tent | 0.67 | 0.67 | | bag | 14.46 | 20.76 | | minibike | 13.59 | 14.06 | | cradle | 73.84 | 81.08 | | oven | 34.97 | 39.1 | | ball | 1.47 | 1.51 | | food | 18.62 | 19.31 | | step | 11.01 | 20.32 | | tank | 16.07 | 16.23 | | trade name | 16.18 | 17.92 | | microwave | 11.58 | 11.97 | | pot | 27.99 | 32.29 | | animal | 37.05 | 37.57 | | bicycle | 43.84 | 51.35 | | lake | 20.16 | 20.17 | | dishwasher | 22.13 | 23.25 | | screen | 32.34 | 32.63 | | blanket | 9.34 | 11.02 | | sculpture | 26.24 | 29.96 | | hood | 26.03 | 26.22 | | sconce | 24.95 | 29.96 | | vase | 32.8 | 48.19 | | traffic light | 19.18 | 21.62 | | tray | 8.42 | 14.19 | | ashcan | 35.89 | 48.04 | | fan | 43.89 | 49.56 | | pier | 5.48 | 6.29 | | crt screen | 6.59 | 21.65 | | plate | 46.8 | 56.56 | | monitor | 29.78 | 33.75 | | bulletin board | 36.28 | 40.9 | | shower | 0.31 | 0.34 | | radiator | 45.87 | 47.09 | | glass | 16.25 | 19.26 | | clock | 35.91 | 44.37 | | flag | 42.76 | 43.7 | +---------------------+-------+-------+ Summary:

+--------+-------+-------+------+ | Scope | mIoU | mAcc | aAcc | +--------+-------+-------+------+ | global | 28.56 | 33.87 | 66.0 | +--------+-------+-------+------+

• Platform:Ubuntu 18.04
• Python version: 3.7.0
• PyTorch version:1.9.1 Did I miss something?

[donglixp]

Hi @yetash , could you share the complete output of the above command? The logging information would be helpful.

[yetash]

Hi @donglixp , I have changed nvidia driver from 470 to 440 and got the right result: +--------+-------+------+-------+ | Scope | mIoU | mAcc | aAcc | +--------+-------+------+-------+ | global | 53.59 | 64.8 | 84.61 | +--------+-------+------+-------+

[bhpfelix]

Hi @yetash, @donglixp, just wanted to follow up on this issue. I seem to have trouble reproducing the performance on Nvidia driver version 470 as well. Unfortunately, I may not be able to tweak the driver version on my machines. Could you please let me know if you have some insights on what could be causing problems for driver version 470 and what could be done to fix it? I am happy to provide my logs and try out things on my end as well.