search

sunoh-kim / pps

Pytorch implementation of the paper 'Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture Diverse Events for Weakly Supervised Temporal Video Grounding' (AAAI2024).
MIT License
13 stars 0 forks source link

Cannot reproduce the paper result #3

Closed fake-warrior8 closed 1 month ago

fake-warrior8 commented 1 month ago

Hi, I reproduce the results using the CPL features, your code and environments. The results are as follows

Charades R1 IoU=0.3, 0.5, 0.7 R5 IoU=0.3 0.5 0.7 69.06 51.49 26.16 99.18 86.23 53.01 (paper) 68.75 50.32 24.89 98.54 87.49 52.94 (your_hyper) 67.80 50.13 25.59 98.48 85.85 51.74 (your_hyper_refact)

Anet R1 IoU=0.1, 0.3, 0.5 R5 IoU=0.1, 0.3, 0.5 81.84 59.29 31.25 95.28 85.54 71.32 (paper) 79.65 52.58 29.12 91.40 71.70 54.27 (your_hyper) 81.57 57.26 31.21 91.97 74.89 58.25 (your_hyper_refact)

  1. I select the best test result using R1 mIoU, which is the same as CPL. I found mIoU's results are positively correlated with these above indicators.
  2. I used A100 but not 4090.
sunoh-kim commented 1 month ago

The best mIoN does not necessarily give the best results. Because the mIoN is average, it can be highly responsive to outliers. Since we have three R1 metrics and three R5 metrics, it would be good to see what results these metrics show. So, please use wandb or tensorboard to see the results. The graph below is the learning curve from the wandb in my environment.

CharadesSTA_Results_R1 CharadesSTA_Results_R5 ActivityNetCaptions_Results_R1 ActivityNetCaptions_Results_R5