search

open-mmlab / mmaction2

OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark
https://mmaction2.readthedocs.io
Apache License 2.0
4.17k stars 1.22k forks source link

test ava can't get same map, why? #1086

Closed liuheng92 closed 3 years ago

liuheng92 commented 3 years ago

If you feel we have help you, give us a STAR! :satisfied:

Notice

There are several common situations in the reimplementation issues as below

  1. Reimplement a model in the model zoo using the provided configs
  2. Reimplement a model in the model zoo on other dataset (e.g., custom datasets)
  3. Reimplement a custom model but all the components are implemented in MMAction2
  4. Reimplement a custom model with new modules implemented by yourself

There are several things to do for different cases as below.

Checklist

  1. I have searched related issues but cannot get the expected help.
  2. The issue has not been fixed in the latest version.

Describe the issue

A clear and concise description of what the problem you meet and what have you done.

Reproduction

  1. What command or script did you run?
python tools/test.py configs/detection/ava/slowfast_kinetics_pretrained_r50_8x8x1_cosine_10e_ava22_rgb.py /data/liuheng/model/spatio_temporal/slowfast_kinetics_pretrained_r50_8x8x1_cosine_10e_ava22_rgb-b987b516.pth --eval mAP --out results.csv
  1. What config dir you run?
configs/detection/ava/slowfast_kinetics_pretrained_r50_8x8x1_cosine_10e_ava22_rgb.py
  1. Did you make any modifications on the code or config? Did you understand what you have modified?
  2. What dataset did you use?

Environment

  1. Please run PYTHONPATH=${PWD}:$PYTHONPATH python mmaction/utils/collect_env.py to collect necessary environment information and paste it here.
  2. You may add addition that may be helpful for locating the problem, such as
    1. How you installed PyTorch [e.g., pip, conda, source]
    2. Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

Results

If applicable, paste the related results here, e.g., what you expect and what you get.

mAP@0.5IOU= 0.2015959468978934
PerformanceByCategory/AP@0.5IOU/bend/bow (at the waist)=    0.3794111613638296
PerformanceByCategory/AP@0.5IOU/crawl=  0.0219318421631896
PerformanceByCategory/AP@0.5IOU/crouch/kneel=   0.2190154107747291
PerformanceByCategory/AP@0.5IOU/dance=  0.5181683502904734
PerformanceByCategory/AP@0.5IOU/fall down=  0.11087082832374898
PerformanceByCategory/AP@0.5IOU/get up= 0.19752443981836204
PerformanceByCategory/AP@0.5IOU/jump/leap=  0.06838310625145752
PerformanceByCategory/AP@0.5IOU/lie/sleep=  0.478338292481998
PerformanceByCategory/AP@0.5IOU/martial art=    0.46829496696242184
PerformanceByCategory/AP@0.5IOU/run/jog=    0.5483911405989155
PerformanceByCategory/AP@0.5IOU/sit=    0.7860971869146263
PerformanceByCategory/AP@0.5IOU/stand=  0.8158033769459013
PerformanceByCategory/AP@0.5IOU/swim=   0.5803478415356426
PerformanceByCategory/AP@0.5IOU/walk=   0.7593141646857874
PerformanceByCategory/AP@0.5IOU/answer phone=   0.7470565693212328
PerformanceByCategory/AP@0.5IOU/brush teeth=    0.00016163843637852977
PerformanceByCategory/AP@0.5IOU/carry/hold (an object)= 0.5576330151503182
PerformanceByCategory/AP@0.5IOU/catch (an object)=  0.0002867517472682873
PerformanceByCategory/AP@0.5IOU/chop=   0.0036451704280475385
PerformanceByCategory/AP@0.5IOU/climb (e.g., a mountain)=   0.08865145043120776
PerformanceByCategory/AP@0.5IOU/clink glass=    0.001533069979500455
PerformanceByCategory/AP@0.5IOU/close (e.g., a door, a box)=    0.15527670644201988
PerformanceByCategory/AP@0.5IOU/cook=   0.014889169393201366
PerformanceByCategory/AP@0.5IOU/cut=    0.02961609898480472
PerformanceByCategory/AP@0.5IOU/dig=    0.0727212786827661
PerformanceByCategory/AP@0.5IOU/dress/put on clothing=  0.12280538320617326
PerformanceByCategory/AP@0.5IOU/drink=  0.2615237919976871
PerformanceByCategory/AP@0.5IOU/drive (e.g., a car, a truck)=   0.5900114289355609
PerformanceByCategory/AP@0.5IOU/eat=    0.29614133937787357
PerformanceByCategory/AP@0.5IOU/enter=  0.054848936954013994
PerformanceByCategory/AP@0.5IOU/exit=   0.00210649738910144
PerformanceByCategory/AP@0.5IOU/extract=    0.0
PerformanceByCategory/AP@0.5IOU/fishing=    0.07957184069800168
PerformanceByCategory/AP@0.5IOU/hit (an object)=    0.07446680601317855
PerformanceByCategory/AP@0.5IOU/kick (an object)=   9.888582589138122e-05
PerformanceByCategory/AP@0.5IOU/lift/pick up=   0.026297920542406997
PerformanceByCategory/AP@0.5IOU/listen (e.g., to music)=    0.00866692672764156
PerformanceByCategory/AP@0.5IOU/open (e.g., a window, a car door)=  0.22771940606663443
PerformanceByCategory/AP@0.5IOU/paint=  0.0014079584101102716
PerformanceByCategory/AP@0.5IOU/play board game=    0.0009136812596669652
PerformanceByCategory/AP@0.5IOU/play musical instrument=    0.39996231605965804
PerformanceByCategory/AP@0.5IOU/play with pets= 0.001215824750286696
PerformanceByCategory/AP@0.5IOU/point to (an object)=   0.0010268637552917812
PerformanceByCategory/AP@0.5IOU/press=  0.004323756379019879
PerformanceByCategory/AP@0.5IOU/pull (an object)=   0.013878239116741313
PerformanceByCategory/AP@0.5IOU/push (an object)=   0.03174632135730881
PerformanceByCategory/AP@0.5IOU/put down=   0.033380213161150536
PerformanceByCategory/AP@0.5IOU/read=   0.28669965205902936
PerformanceByCategory/AP@0.5IOU/ride (e.g., a bike, a car, a horse)=    0.45511476174061494
PerformanceByCategory/AP@0.5IOU/row boat=   0.03287746069852424
PerformanceByCategory/AP@0.5IOU/sail boat=  0.1319843453373496
PerformanceByCategory/AP@0.5IOU/shoot=  0.055577517720488294
PerformanceByCategory/AP@0.5IOU/shovel= 0.20433942843531064
PerformanceByCategory/AP@0.5IOU/smoke=  0.20459064199487978
PerformanceByCategory/AP@0.5IOU/stir=   0.006029776553388519
PerformanceByCategory/AP@0.5IOU/take a photo=   0.014536299657359458
PerformanceByCategory/AP@0.5IOU/text on/look at a cellphone=    0.06536469477232087
PerformanceByCategory/AP@0.5IOU/throw=  0.02540799181965103
PerformanceByCategory/AP@0.5IOU/touch (an object)=  0.32652192007998754
PerformanceByCategory/AP@0.5IOU/turn (e.g., a screwdriver)= 0.008081544522191998
PerformanceByCategory/AP@0.5IOU/watch (e.g., TV)=   0.2631289265695224
PerformanceByCategory/AP@0.5IOU/work on a computer= 0.05864314772710921
PerformanceByCategory/AP@0.5IOU/write=  0.1427149891953294
PerformanceByCategory/AP@0.5IOU/fight/hit (a person)=   0.44859908262399345
PerformanceByCategory/AP@0.5IOU/give/serve (an object) to (a person)=   0.07400330739672253
PerformanceByCategory/AP@0.5IOU/grab (a person)=    0.07822650024972234
PerformanceByCategory/AP@0.5IOU/hand clap=  0.35221877005999486
PerformanceByCategory/AP@0.5IOU/hand shake= 0.1269389643831672
PerformanceByCategory/AP@0.5IOU/hand wave=  0.03296805126067947
PerformanceByCategory/AP@0.5IOU/hug (a person)= 0.19509946126454697
PerformanceByCategory/AP@0.5IOU/kick (a person)=    0.004435801264502675
PerformanceByCategory/AP@0.5IOU/kiss (a person)=    0.3112509902252728
PerformanceByCategory/AP@0.5IOU/lift (a person)=    0.04049703109874648
PerformanceByCategory/AP@0.5IOU/listen to (a person)=   0.5768687870213601
PerformanceByCategory/AP@0.5IOU/play with kids= 0.0015044030943416368
PerformanceByCategory/AP@0.5IOU/push (another person)=  0.053615943491820965
PerformanceByCategory/AP@0.5IOU/sing to (e.g., self, a person, a group)=    0.18274009150937548
PerformanceByCategory/AP@0.5IOU/take (an object) from (a person)=   0.05088378012805926
PerformanceByCategory/AP@0.5IOU/talk to (e.g., self, a person, a group)=    0.7919251793517121
PerformanceByCategory/AP@0.5IOU/watch (a person)=   0.6688091424331698

mAP@0.5IOU   0.2016
mAP@0.5IOU: 0.2016

Issue fix

If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

kennymckormick commented 3 years ago

Hi, we are trying to reproduce this problem. Once we get a conclusion, we will tell you asap.

liuheng92 commented 3 years ago

Hi, we are trying to reproduce this problem. Once we get a conclusion, we will tell you asap.

ok, thx. Look forward to your kind reply.

kennymckormick commented 3 years ago

Hello, we find that we have used the wrong label_file in AVA2.2 configs, which report the mAP on 80 classes instead of 60 (The performance of AVA models should be evaluated on 60 classes). We have fixed it in PR #1081.

liuheng92 commented 3 years ago

Hello, we find that we have used the wrong label_file in AVA2.2 configs, which report the mAP on 80 classes instead of 60 (The performance of AVA models should be evaluated on 60 classes). We have fixed it in PR #1081.

got it! thx