Can you share a trained ucf24 weight file?

[wjtan99]

The directory ucf24 google drive is empty. I am training now with 4 x 1080TI GPUs on ucf24. It runs very slowly, "Itration [1/10]006291/121470" took almost 2 hours. At this speed, 1 epoch will take 40 hours. I do not know what is wrong and why it runs so slowly.
Thanks.

[gurkirt]

I did most of the experiments at old uni lab computers, I do not have access to those anymore. I will ask someone about it.

Can you check GPU utilisations? also check the number of workers being used? The disk read could be an issue, if have a small SSD drive ~24-40GB that could solve that, since ucf24 dataset requires very little space.

[wjtan99]

Thanks for your reply. It runs faster later. In the past nearly 30 hours, it finishes 6 epochs. So not too bad. I believe the reason is that my data is saved on an HDD, not an SSD.

[wjtan99]

My validation results at the 6th epoch are as follows,

[INFO: val.py: 121]: Evaluating detections for epoch number 6 [INFO: evaluation.py: 107]: Evaluating for 79488 frames [INFO: evaluation.py: 171]: mean ap 88.59203 [INFO: evaluation.py: 107]: Evaluating for 79488 frames [INFO: evaluation.py: 171]: mean ap 83.210945 [INFO: evaluation.py: 208]: Evaluating for 79488 frames [INFO: train.py: 170]: action_ness : 79594.0 : 113765 : 88.59203340705088 [INFO: train.py: 172]: action_ness MEANAP:::=> 88.59203 [INFO: train.py: 170]: Basketball : 954.0 : 95486 : 59.51195499250714 [INFO: train.py: 170]: BasketballDunk : 796.0 : 95666 : 59.78202947168123 [INFO: train.py: 170]: Biking : 5059.0 : 161896 : 94.40393395221413 [INFO: train.py: 170]: CliffDiving : 1045.0 : 83174 : 83.1584996550214 [INFO: train.py: 170]: CricketBowling : 957.0 : 103464 : 77.74941856822169 [INFO: train.py: 170]: Diving : 2129.0 : 83262 : 87.40305250413103 [INFO: train.py: 170]: Fencing : 4246.0 : 96746 : 95.09716883209339 [INFO: train.py: 170]: FloorGymnastics : 2784.0 : 67934 : 97.81310098666995 [INFO: train.py: 170]: GolfSwing : 2378.0 : 65753 : 79.03662979501055 [INFO: train.py: 170]: HorseRiding : 4562.0 : 119466 : 97.53178202916743 [INFO: train.py: 170]: IceDancing : 9465.0 : 92472 : 79.20677371037966 [INFO: train.py: 170]: LongJump : 2510.0 : 112139 : 88.82021353393093 [INFO: train.py: 170]: PoleVault : 3191.0 : 127231 : 79.73631596568242 [INFO: train.py: 170]: RopeClimbing : 2848.0 : 74503 : 98.1229738185184 [INFO: train.py: 170]: SalsaSpin : 7750.0 : 103606 : 92.16125624555819 [INFO: train.py: 170]: SkateBoarding : 2408.0 : 90512 : 96.76801128668181 [INFO: train.py: 170]: Skiing : 4064.0 : 59137 : 91.81366633028539 [INFO: train.py: 170]: Skijet : 3199.0 : 74643 : 94.71636044769825 [INFO: train.py: 170]: SoccerJuggling : 5738.0 : 110843 : 98.6223570434686 [INFO: train.py: 170]: Surfing : 3084.0 : 76499 : 93.65102430087813 [INFO: train.py: 170]: TennisSwing : 1346.0 : 87748 : 36.174880175618576 [INFO: train.py: 170]: TrampolineJumping : 5183.0 : 91844 : 86.01334420915046 [INFO: train.py: 170]: VolleyballSpiking : 284.0 : 106229 : 35.68569960181739 [INFO: train.py: 170]: WalkingWithDog : 3614.0 : 122703 : 94.08224531651302 [INFO: train.py: 172]: action MEANAP:::=> 83.21095 [INFO: train.py: 170]: Non_action : 13055 : 79488 : 89.60893318379097 [INFO: train.py: 170]: Basketball : 944 : 79488 : 81.64805368061556 [INFO: train.py: 170]: BasketballDunk : 796 : 79488 : 82.49892382914202 [INFO: train.py: 170]: Biking : 4170 : 79488 : 99.71649158868784 [INFO: train.py: 170]: CliffDiving : 1029 : 79488 : 93.43391564329245 [INFO: train.py: 170]: CricketBowling : 949 : 79488 : 89.16154496935629 [INFO: train.py: 170]: Diving : 2129 : 79488 : 97.02275698795985 [INFO: train.py: 170]: Fencing : 2080 : 79488 : 99.5068332237833 [INFO: train.py: 170]: FloorGymnastics : 2784 : 79488 : 99.36656090254712 [INFO: train.py: 170]: GolfSwing : 2378 : 79488 : 93.13977583900498 [INFO: train.py: 170]: HorseRiding : 4424 : 79488 : 99.03349999274505 [INFO: train.py: 170]: IceDancing : 5120 : 79488 : 99.65199187022361 [INFO: train.py: 170]: LongJump : 2510 : 79488 : 98.90832243869242 [INFO: train.py: 170]: PoleVault : 3189 : 79488 : 92.8485713067943 [INFO: train.py: 170]: RopeClimbing : 2848 : 79488 : 98.46269087019617 [INFO: train.py: 170]: SalsaSpin : 4263 : 79488 : 98.39607277567902 [INFO: train.py: 170]: SkateBoarding : 2408 : 79488 : 98.2783613425767 [INFO: train.py: 170]: Skiing : 4064 : 79488 : 96.57675081715995 [INFO: train.py: 170]: Skijet : 3199 : 79488 : 99.8667859604034 [INFO: train.py: 170]: SoccerJuggling : 5738 : 79488 : 99.27125694603691 [INFO: train.py: 170]: Surfing : 3084 : 79488 : 98.57784858307976 [INFO: train.py: 170]: TennisSwing : 1346 : 79488 : 73.03256821510884 [INFO: train.py: 170]: TrampolineJumping : 3165 : 79488 : 98.90258271188783 [INFO: train.py: 170]: VolleyballSpiking : 282 : 79488 : 39.71736836340114 [INFO: train.py: 170]: WalkingWithDog : 3534 : 79488 : 96.60830356336312 [INFO: train.py: 170]: FRAME Mean AP:: 92.53 [INFO: train.py: 172]: ego_action MEANAP:::=> 92.52947 [INFO: train.py: 188]: Validation TIME::: 2524.965

However, in Table 5 of your paper, you have frame mAP 75.2, which is a lot lower than the numbers I have. So is there anything wrong with my understanding? BTW, autonomous driving is new to me. You have action-ness mAP, action mAP, and ego_action mAP. Can you explain briefly what the differences are?

Thanks.

[gurkirt]

Those are probably validation results not testing. Validation is done on subset if I rember right. run test_det.py

[gurkirt]

Can you report what are the results when you run main.py with --MODE gen_dets ?

[wjtan99]

I stopped it. It took too long to run. Multiple GPU does not work. BTW, do you know the paper YOWO? The UCF frame mAP is 80.x. I found a bug in his code, and my test result is 83.1 on the test set.

[wjtan99]

My data is saved on HDD, not on SSD. But still, it is too slow to take a week to run on the test set. I changed it to multiple GPUs but only one GPU is used. I do not have time to track down why.

[gurkirt]

I think testing should work on multiple GPU. Might be worth spending 100 USD on SSD. Can you afford to leave the test running for a week? I can't think of a reason why it wouldn't use all the GPUs. Maybe some Cuda related environment variable causing the issue. Try CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --MODE gen_dets.

Yes, I know that works. I haven't tried their code. Seems like a backward step from 3D-retinanet. I tried pyslowfast on UCF24 with YOLOv5 proposals. it wasn't that great.

I was waiting for https://github.com/ShoufaChen/WOO, but it seems the author is busy there. WOO seems similar to 3D-RetinaNet. Would love to try light version of 3D-RetinaNet, i.e. 3D-YOLOv5. Would be happy to accept that PR.

[wjtan99]

I already tried CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py --MODE gen_dets. The multiple GPUs worked on training but not on testing. I think it should be something simple.

YOWO backbone is a mixture of 3D Resnet and 2D YOLOv3. It works great. The frame mAP on UCF24 is >80. I think your paper should cite YOWO. If you want to publish, the reviewer will ask for it.

I know the paper of WOO. I do not think they will ever open source their code.

Can you give me your email so we can communicate more? Thanks. Or you can contact me at wjtan99@gmail.com.

[wjtan99]

I used SSD before. I was an HDD/SDD chip design engineer for 13 years before I work on machine learning now. SSD can easily crash, particularly when we change data frequently, which is what I do.

[gurkirt]

We wrote paper before YOWO, at start of 2021, it is already published https://www.computer.org/csdl/journal/tp/5555/01/09712346/1AZL0P4dL1e

[gurkirt]

Working great alone is not good enough, you need to have simplicity and useability as well. Which is better in WOO and 3Dretinenet rather than YOWO. Don't take me wrong, YOWO is good but I don't like things that are overly complicated, same goes to interaction head in WOO. Seems shortsighted where it be good for AVA dataset but unnecessary for sports-based dataset like UCF24 and Multisports.

[wjtan99]

You must test YOWO yourself. It is not complicated. It runs very fast. The paper states that as a contribution, see Table 9 of https://arxiv.org/pdf/1911.06644.pdf. It is faster than your early paper with 2D-SSD.

[wjtan99]

On the same machine, it only takes less than 2 hours to process all the test videos with 4 1080Ti GPUs. I will double-check if the test video list (testlist_video.txt) is the same as in your 3D-RetinaNet. He uses the same dataset your cleaned up in your earlier repo.

[wjtan99]

I just checked your annotation file 'pyannot_with_class_names.pkl' and your code. There is no separate test set, the test set is the same as the validation set. Here is where the dataset is defined in data/datasets.py,

    for videoname in sorted(database.keys()):

        if debug and True:
            print("videoname = ", videoname) 

        if videoname in self.videos_done: 
            continue 

        is_part = 1
        if 'train' in self.SUBSETS and videoname not in self.trainvideos:
            continue
        elif 'test' in self.SUBSETS and videoname in self.trainvideos:
            continue

You only have two datasets defined in main.py: train_dataset = VideoDataset(args, train=True, skip_step=train_skip_step, transform=train_transform) val_dataset = VideoDataset(args, train=False, transform=val_transform, skip_step=skip_step, full_test=full_test)

So the validation results in the middle of training are meaningful test results. After 6 epochs I got results better than the ones you published in your paper. And there is no code test_det.py in your repo.

[wjtan99]

Found out one issue. With the command you give on your front page, the gen_dets run on all videos, including training and testing videos. That is why it takes so long. We should add --TEST_SUBSET=test. I already added TEST_BATCH_SIZE=4 in my previous test.

    if 'train' in self.SUBSETS and videoname not in self.trainvideos:
        continue
    elif 'test' in self.SUBSETS and videoname in self.trainvideos:
        continue

When "--TEST_SUBSET=test" is not added, the above two lines do not run, and all videos are processed.

[wjtan99]

The multiple GPU may be a Pytorch version problem. I am not sure. Anyway, I made some minor changes, now the gen_dets is running on 4 GPUs now. I will let you know the results later.

[gurkirt]

There is no test set in ucf24, the subset is still val, but during validation, at training time it is further subsampled.

[wjtan99]

OK. Got it. My full-test evaluation is still running. It will take a few more hours. I will post the results once I have them.

[wjtan99]

Here are the results of eval_framewise_dets( ). I do not know which number is the one reported in Table 5. And this is after 6 epoch training.

[INFO: evaluation.py: 545]: Evaluating frames for datasets ucf24 [INFO: evaluation.py: 563]: Time taken to load for evaluation 7.093666225206107 [INFO: evaluation.py: 208]: Evaluating for 159289 frames [INFO: evaluation.py: 578]: Non_action : 20762 : 159289 : 70.35434605863412 [INFO: evaluation.py: 578]: Basketball : 1323 : 159289 : 55.86666277852309 [INFO: evaluation.py: 578]: BasketballDunk : 1409 : 159289 : 86.95094664015275 [INFO: evaluation.py: 578]: Biking : 8870 : 159289 : 98.24389269394766 [INFO: evaluation.py: 578]: CliffDiving : 2194 : 159289 : 86.49736346764362 [INFO: evaluation.py: 578]: CricketBowling : 1463 : 159289 : 58.290228986443324 [INFO: evaluation.py: 578]: Diving : 5168 : 159289 : 94.88568855238293 [INFO: evaluation.py: 578]: Fencing : 4410 : 159289 : 98.4784377529801 [INFO: evaluation.py: 578]: FloorGymnastics : 5519 : 159289 : 95.73548383439052 [INFO: evaluation.py: 578]: GolfSwing : 4344 : 159289 : 68.29019395025277 [INFO: evaluation.py: 578]: HorseRiding : 8491 : 159289 : 98.19545504842783 [INFO: evaluation.py: 578]: IceDancing : 11434 : 159289 : 98.22827034864675 [INFO: evaluation.py: 578]: LongJump : 4480 : 159289 : 95.41562959232529 [INFO: evaluation.py: 578]: PoleVault : 7114 : 159289 : 72.48295404544423 [INFO: evaluation.py: 578]: RopeClimbing : 5691 : 159289 : 92.23526319581569 [INFO: evaluation.py: 578]: SalsaSpin : 9516 : 159289 : 94.34762568113638 [INFO: evaluation.py: 578]: SkateBoarding : 4832 : 159289 : 88.50440638067509 [INFO: evaluation.py: 578]: Skiing : 8976 : 159289 : 83.95897579327931 [INFO: evaluation.py: 578]: Skijet : 7319 : 159289 : 99.35816749726666 [INFO: evaluation.py: 578]: SoccerJuggling : 12271 : 159289 : 95.47832320233263 [INFO: evaluation.py: 578]: Surfing : 5851 : 159289 : 96.41417088426883 [INFO: evaluation.py: 578]: TennisSwing : 2876 : 159289 : 41.57050491180705 [INFO: evaluation.py: 578]: TrampolineJumping : 6666 : 159289 : 97.32566619189322 [INFO: evaluation.py: 578]: VolleyballSpiking : 1131 : 159289 : 56.49974181322741 [INFO: evaluation.py: 578]: WalkingWithDog : 7179 : 159289 : 83.86893886508591 [INFO: evaluation.py: 578]: FRAME Mean AP:: 84.30 [INFO: evaluation.py: 626]: MAP:: 84.29909352667933 [INFO: evaluation.py: 626]: MAP:: 22.376298904418945

[INFO: evaluation.py: 626]: MAP:: 18.945189223935206 [INFO: evaluation.py: 628]: Time taken to complete evaluation 271.66151784406975 [INFO: gen_dets.py: 375]:

Results for test & action_ness

[INFO: gen_dets.py: 379]: action_ness : 170580 : 5567603 : 22.376298904418945 : 95.04982829093933 [INFO: gen_dets.py: 379]: Mean AP:: 22.38 mean Recall 95.05 [INFO: gen_dets.py: 375]:

Results for test & action

[INFO: gen_dets.py: 379]: Basketball : 1323 : 2103555 : 11.37794777750969 : 97.73242473602295 [INFO: gen_dets.py: 379]: BasketballDunk : 1409 : 1218073 : 11.415847390890121 : 86.79915070533752 [INFO: gen_dets.py: 379]: Biking : 11911 : 1892836 : 23.193584382534027 : 90.21912813186646 [INFO: gen_dets.py: 379]: CliffDiving : 2232 : 1437192 : 14.296434819698334 : 90.41218757629395 [INFO: gen_dets.py: 379]: CricketBowling : 1562 : 2136839 : 13.477268815040588 : 92.31753945350647 [INFO: gen_dets.py: 379]: Diving : 5168 : 1193781 : 25.08368492126465 : 96.63312435150146 [INFO: gen_dets.py: 379]: Fencing : 9564 : 1516513 : 21.843525767326355 : 90.74655175209045 [INFO: gen_dets.py: 379]: FloorGymnastics : 5519 : 1265288 : 24.08924400806427 : 97.4995493888855 [INFO: gen_dets.py: 379]: GolfSwing : 4344 : 1167316 : 10.285208374261856 : 100.0 [INFO: gen_dets.py: 379]: HorseRiding : 8496 : 1824711 : 28.45887839794159 : 99.30555820465088 [INFO: gen_dets.py: 379]: IceDancing : 22474 : 1916024 : 42.81310439109802 : 98.99884462356567 [INFO: gen_dets.py: 379]: LongJump : 4480 : 2057967 : 14.72005695104599 : 84.66517925262451 [INFO: gen_dets.py: 379]: PoleVault : 7114 : 1999204 : 9.892335534095764 : 90.0477945804596 [INFO: gen_dets.py: 379]: RopeClimbing : 5691 : 1334938 : 21.014994382858276 : 99.80671405792236 [INFO: gen_dets.py: 379]: SalsaSpin : 17131 : 1489375 : 22.05546796321869 : 98.51731061935425 [INFO: gen_dets.py: 379]: SkateBoarding : 4832 : 1910520 : 24.839815497398376 : 99.44122433662415 [INFO: gen_dets.py: 379]: Skiing : 8976 : 1161361 : 18.103520572185516 : 82.26381540298462 [INFO: gen_dets.py: 379]: Skijet : 7319 : 1106284 : 17.374128103256226 : 96.9941258430481 [INFO: gen_dets.py: 379]: SoccerJuggling : 12271 : 1885668 : 24.937599897384644 : 99.54363703727722 [INFO: gen_dets.py: 379]: Surfing : 5851 : 1501157 : 23.20406138896942 : 95.89813947677612 [INFO: gen_dets.py: 379]: TennisSwing : 2876 : 1537559 : 7.38600492477417 : 96.24478220939636 [INFO: gen_dets.py: 379]: TrampolineJumping : 11713 : 1480148 : 12.490403652191162 : 88.38896751403809 [INFO: gen_dets.py: 379]: VolleyballSpiking : 1131 : 1668585 : 7.008560001850128 : 88.77099752426147 [INFO: gen_dets.py: 379]: WalkingWithDog : 7193 : 1706130 : 25.322863459587097 : 96.62171602249146 [INFO: gen_dets.py: 379]: Mean AP:: 18.95 mean Recall 94.08 [INFO: gen_dets.py: 375]:

Results for test & frame_actions

[INFO: gen_dets.py: 379]: Non_action : 20762 : 159289 : 70.35434605863412 [INFO: gen_dets.py: 379]: Basketball : 1323 : 159289 : 55.86666277852309 [INFO: gen_dets.py: 379]: BasketballDunk : 1409 : 159289 : 86.95094664015275 [INFO: gen_dets.py: 379]: Biking : 8870 : 159289 : 98.24389269394766 [INFO: gen_dets.py: 379]: CliffDiving : 2194 : 159289 : 86.49736346764362 [INFO: gen_dets.py: 379]: CricketBowling : 1463 : 159289 : 58.290228986443324 [INFO: gen_dets.py: 379]: Diving : 5168 : 159289 : 94.88568855238293 [INFO: gen_dets.py: 379]: Fencing : 4410 : 159289 : 98.4784377529801 [INFO: gen_dets.py: 379]: FloorGymnastics : 5519 : 159289 : 95.73548383439052 [INFO: gen_dets.py: 379]: GolfSwing : 4344 : 159289 : 68.29019395025277 [INFO: gen_dets.py: 379]: HorseRiding : 8491 : 159289 : 98.19545504842783 [INFO: gen_dets.py: 379]: IceDancing : 11434 : 159289 : 98.22827034864675 [INFO: gen_dets.py: 379]: LongJump : 4480 : 159289 : 95.41562959232529 [INFO: gen_dets.py: 379]: PoleVault : 7114 : 159289 : 72.48295404544423 [INFO: gen_dets.py: 379]: RopeClimbing : 5691 : 159289 : 92.23526319581569 [INFO: gen_dets.py: 379]: SalsaSpin : 9516 : 159289 : 94.34762568113638 [INFO: gen_dets.py: 379]: SkateBoarding : 4832 : 159289 : 88.50440638067509 [INFO: gen_dets.py: 379]: Skiing : 8976 : 159289 : 83.95897579327931 [INFO: gen_dets.py: 379]: Skijet : 7319 : 159289 : 99.35816749726666 [INFO: gen_dets.py: 379]: SoccerJuggling : 12271 : 159289 : 95.47832320233263 [INFO: gen_dets.py: 379]: Surfing : 5851 : 159289 : 96.41417088426883 [INFO: gen_dets.py: 379]: TennisSwing : 2876 : 159289 : 41.57050491180705 [INFO: gen_dets.py: 379]: TrampolineJumping : 6666 : 159289 : 97.32566619189322 [INFO: gen_dets.py: 379]: VolleyballSpiking : 1131 : 159289 : 56.49974181322741 [INFO: gen_dets.py: 379]: WalkingWithDog : 7179 : 159289 : 83.86893886508591 [INFO: gen_dets.py: 379]: FRAME Mean AP:: 84.30 [INFO: gen_dets.py: 379]: Mean AP:: 84.30 mean Recall 1.00

[wjtan99]

Here are the results of build_eval_tubes( ). Again this is after 6 epoch training.

[INFO: evaluation.py: 347]: Evaluating tubes for datasets ucf24 [INFO: evaluation.py: 348]: GT FILE:: ./ucf24/pyannot_with_class_names.pkl [INFO: evaluation.py: 349]: Result File:: ./output/ucf24/cache/resnet50I3D512-Pkinetics-b4s8x1x1-ucf24tn-h3x3x3//tubes-10-08-8000-20-score-25-4/tubes_indiv_0.pkl [INFO: evaluation.py: 374]: Evaluating action 24 [INFO: evaluation.py: 404]: MAP:: 22.879461212141905 [INFO: tubes.py: 79]:

Results for test & action @ 0.20 stiou

[INFO: tubes.py: 87]: Basketball : 35 : 19411 : 10.160548239946365 : 91.42857193946838 [INFO: tubes.py: 87]: BasketballDunk : 38 : 16662 : 18.659785389900208 : 97.36841917037964 [INFO: tubes.py: 87]: Biking : 67 : 19004 : 26.234880089759827 : 82.08954930305481 [INFO: tubes.py: 87]: CliffDiving : 40 : 15404 : 33.78438651561737 : 94.9999988079071 [INFO: tubes.py: 87]: CricketBowling : 38 : 22470 : 27.22591459751129 : 92.1052634716034 [INFO: tubes.py: 87]: Diving : 45 : 16000 : 45.40447294712067 : 100.0 [INFO: tubes.py: 87]: Fencing : 81 : 19264 : 0.028999766800552607 : 4.938271641731262 [INFO: tubes.py: 87]: FloorGymnastics : 36 : 17212 : 0.17295002471655607 : 5.55555559694767 [INFO: tubes.py: 87]: GolfSwing : 39 : 10242 : 1.3193324208259583 : 20.512820780277252 [INFO: tubes.py: 87]: HorseRiding : 51 : 16907 : 0.0 : 0.0 [INFO: tubes.py: 87]: IceDancing : 107 : 17613 : 62.7072274684906 : 89.71962332725525 [INFO: tubes.py: 87]: LongJump : 38 : 22461 : 27.648836374282837 : 92.1052634716034 [INFO: tubes.py: 87]: PoleVault : 43 : 24981 : 16.996274888515472 : 97.67441749572754 [INFO: tubes.py: 87]: RopeClimbing : 34 : 17085 : 29.712557792663574 : 100.0 [INFO: tubes.py: 87]: SalsaSpin : 210 : 13672 : 20.1000839471817 : 80.95238208770752 [INFO: tubes.py: 87]: SkateBoarding : 32 : 18217 : 42.45055317878723 : 100.0 [INFO: tubes.py: 87]: Skiing : 40 : 12113 : 38.22225332260132 : 89.99999761581421 [INFO: tubes.py: 87]: Skijet : 29 : 14277 : 18.194912374019623 : 93.1034505367279 [INFO: tubes.py: 87]: SoccerJuggling : 41 : 14246 : 33.42943787574768 : 95.12194991111755 [INFO: tubes.py: 87]: Surfing : 52 : 16392 : 32.300350069999695 : 94.2307710647583 [INFO: tubes.py: 87]: TennisSwing : 66 : 14914 : 8.534009009599686 : 62.12121248245239 [INFO: tubes.py: 87]: TrampolineJumping : 76 : 15565 : 8.038446307182312 : 86.84210777282715 [INFO: tubes.py: 87]: VolleyballSpiking : 37 : 20700 : 9.503355622291565 : 89.18918967247009 [INFO: tubes.py: 87]: WalkingWithDog : 40 : 13299 : 38.27750086784363 : 92.5000011920929 [INFO: tubes.py: 87]: Mean AP:: 22.88 mean Recall 77.19 [INFO: evaluation.py: 347]: Evaluating tubes for datasets ucf24 [INFO: evaluation.py: 348]: GT FILE:: ./ucf24/pyannot_with_class_names.pkl [INFO: evaluation.py: 349]: Result File:: ./output/ucf24/cache/resnet50I3D512-Pkinetics-b4s8x1x1-ucf24tn-h3x3x3//tubes-10-08-8000-20-score-25-4/tubes_indiv_0.pkl [INFO: evaluation.py: 374]: Evaluating action 24 [INFO: evaluation.py: 404]: MAP:: 15.678471503391242 [INFO: tubes.py: 79]:

Results for test & action @ 0.50 stiou

[INFO: tubes.py: 87]: Basketball : 35 : 19411 : 0.5247263237833977 : 48.571428656578064 [INFO: tubes.py: 87]: BasketballDunk : 38 : 16662 : 6.8986475467681885 : 52.63158082962036 [INFO: tubes.py: 87]: Biking : 67 : 19004 : 19.88183856010437 : 58.20895433425903 [INFO: tubes.py: 87]: CliffDiving : 40 : 15404 : 17.933548986911774 : 72.50000238418579 [INFO: tubes.py: 87]: CricketBowling : 38 : 22470 : 2.6141753420233727 : 44.736841320991516 [INFO: tubes.py: 87]: Diving : 45 : 16000 : 38.357704877853394 : 91.11111164093018 [INFO: tubes.py: 87]: Fencing : 81 : 19264 : 0.0 : 0.0 [INFO: tubes.py: 87]: FloorGymnastics : 36 : 17212 : 0.0 : 0.0 [INFO: tubes.py: 87]: GolfSwing : 39 : 10242 : 0.0 : 0.0 [INFO: tubes.py: 87]: HorseRiding : 51 : 16907 : 0.0 : 0.0 [INFO: tubes.py: 87]: IceDancing : 107 : 17613 : 48.63781929016113 : 77.57009267807007 [INFO: tubes.py: 87]: LongJump : 38 : 22461 : 16.03659689426422 : 71.05262875556946 [INFO: tubes.py: 87]: PoleVault : 43 : 24981 : 8.617711067199707 : 67.44186282157898 [INFO: tubes.py: 87]: RopeClimbing : 34 : 17085 : 29.712557792663574 : 100.0 [INFO: tubes.py: 87]: SalsaSpin : 210 : 13672 : 1.3975498266518116 : 19.523809850215912 [INFO: tubes.py: 87]: SkateBoarding : 32 : 18217 : 42.45055317878723 : 100.0 [INFO: tubes.py: 87]: Skiing : 40 : 12113 : 36.704039573669434 : 77.49999761581421 [INFO: tubes.py: 87]: Skijet : 29 : 14277 : 10.615897178649902 : 75.86206793785095 [INFO: tubes.py: 87]: SoccerJuggling : 41 : 14246 : 33.232566714286804 : 92.68292784690857 [INFO: tubes.py: 87]: Surfing : 52 : 16392 : 23.694564402103424 : 69.2307710647583 [INFO: tubes.py: 87]: TennisSwing : 66 : 14914 : 0.1415626611560583 : 6.060606241226196 [INFO: tubes.py: 87]: TrampolineJumping : 76 : 15565 : 1.027736347168684 : 32.894736528396606 [INFO: tubes.py: 87]: VolleyballSpiking : 37 : 20700 : 5.137879401445389 : 56.75675868988037 [INFO: tubes.py: 87]: WalkingWithDog : 40 : 13299 : 32.665640115737915 : 80.0000011920929 [INFO: tubes.py: 87]: Mean AP:: 15.68 mean Recall 53.93 [INFO: tubes.py: 113]:	class	stiou 0.20	stiou 0.50
mAP	22.9/77.2	15.7/53.9
Basketball	10.2/91.4	0.5/48.6
BasketballDunk	18.7/97.4	6.9/52.6
Biking	26.2/82.1	19.9/58.2
CliffDiving	33.8/95.0	17.9/72.5
CricketBowling	27.2/92.1	2.6/44.7
Diving	45.4/100.0	38.4/91.1
Fencing	0.0/4.9	0.0/0.0
FloorGymnastics	0.2/5.6	0.0/0.0
GolfSwing	1.3/20.5	0.0/0.0
HorseRiding	0.0/0.0	0.0/0.0
IceDancing	62.7/89.7	48.6/77.6
LongJump	27.6/92.1	16.0/71.1
PoleVault	17.0/97.7	8.6/67.4
RopeClimbing	29.7/100.0	29.7/100.0
SalsaSpin	20.1/81.0	1.4/19.5
SkateBoarding	42.5/100.0	42.5/100.0
Skiing	38.2/90.0	36.7/77.5
Skijet	18.2/93.1	10.6/75.9
SoccerJuggling	33.4/95.1	33.2/92.7
Surfing	32.3/94.2	23.7/69.2
TennisSwing	8.5/62.1	0.1/6.1
TrampolineJumping	8.0/86.8	1.0/32.9
VolleyballSpiking	9.5/89.2	5.1/56.8
WalkingWithDog	38.3/92.5	32.7/80.0

[gurkirt]

Honestly, what is going on here? I will rerun it myself on my new machine and let you know the results. Results look too bad to be true. Can list what all changes that you made?

[wjtan99]

I did not make changes to the algorithm itself. I guess the biggest factor is that I only trained the model for 6 epochs. How many epochs should I train to get good performance?

[gurkirt]

CUDA_VISIBLE_DEVICES=0,1,2,3 python main.py /home/user/ /home/user/ /home/user/kinetics-pt/ --MODE=train --ARCH=resnet50 --MODEL_TYPE=I3D --DATASET=ucf24 --TRAIN_SUBSETS=train --VAL_SUBSETS=val --SEQ_LEN=8 --TEST_SEQ_LEN=8 --BATCH_SIZE=4 --LR=0.00245 --MILESTONES=6,8 --MAX_EPOCHS=10

[gurkirt]

I went to have look at YOWO, performance on AVA is lacking by quite a bit almost 2/3rd from slowfast.