search

Pilhyeon / WTAL-Uncertainty-Modeling

Official Pytorch Implementation of 'Weakly-supervised Temporal Action Localization by Uncertainty Modeling' (AAAI-21)
MIT License
123 stars 10 forks source link

Reproducing the results on ActivityNet #6

Open tobbysong opened 3 years ago

tobbysong commented 3 years ago

Hi @Pilhyeon, thanks for your great work! Now I'm following your BMUE and have some trouble in reproducing the results on ActivityNet dataset.

I have tried to do some experiments on ActivityNet v1.2. I downloaded the I3D features provided by this link and adapt them to BMUE format. The following are some of my results. All experiments run for 6k epochs, the results are showed in the form of "(average_mAP, Test_acc)":

According to Sec 4.1 in your arXiv paper, T=50, so I set num_segments as 50 and do the following experiments:

Besides, I also tried to change "act_thresh_magnitudes", "NMS thresh", "alpha", "_lambda & gamma in get_proposal_oic()", etc. The results don't seem to get better: the test accuracy is around 0.4 and the average_mAP is very low. It's hard for me to find the best settings. Could you please share your params settings on ActivityNet 1.2 & 1.3 datasets? Or give me some advice on which params to change?

Looking forward to your reply and I'd be glad to cite your excellent work. Thanks!

Pilhyeon commented 3 years ago

First of all, I apologize for the missing details about the experimental settings on ActivityNet. Since I extracted the features for ActivityNet by myself, they might be different from those you used. For example, the sampling rate of the videos or the pre-trained I3D models could be different. I used the original fps of the videos and the pre-trained models from this repo. For the hyperparameters, the batch size, the margin, alpha, (r_act, r_bkg), and the NMS threshold are set to 512, 250, 1e-4, (5, 50), and 0.8, respectively. I hope this would help.

xumh-9 commented 3 years ago

Can you share the feature of act1.2?

cecilia930426 commented 3 years ago

Thanks for your wonderful work. Could you please share your features in Activitynet 1.3?

P-DX commented 3 years ago

Hi, have you got the better result after you set the hyperparameters as mentioned? Thanks.

sauradip commented 3 years ago

Hi @Pilhyeon, thanks for your great work! Now I'm following your BMUE and have some trouble in reproducing the results on ActivityNet dataset.

I have tried to do some experiments on ActivityNet v1.2. I downloaded the I3D features provided by this link and adapt them to BMUE format. The following are some of my results. All experiments run for 6k epochs, the results are showed in the form of "(average_mAP, Test_acc)":

  • [no params changed] (0.0390, 0.43)

According to Sec 4.1 in your arXiv paper, T=50, so I set num_segments as 50 and do the following experiments:

  • [num_segments: 50] (0.0275, 0.31)

    • [class_th: 0.1] (0.0332, 0.36)
    • [class_th: 0.1, act_thresh_cas: np.arange(0.0, 0.15, 0.015)] (0.0400, 0.36)

Besides, I also tried to change "act_thresh_magnitudes", "NMS thresh", "alpha", "_lambda & gamma in get_proposal_oic()", etc. The results don't seem to get better: the test accuracy is around 0.4 and the average_mAP is very low. It's hard for me to find the best settings. Could you please share your params settings on ActivityNet 1.2 & 1.3 datasets? Or give me some advice on which params to change?

Looking forward to your reply and I'd be glad to cite your excellent work. Thanks!

Hi toby ,

Can you kindly share your version of the repo for ActivityNet dataset in this email : sauradipnag95@gmail.com if possible ?

liming-ai commented 3 years ago

Hi @Pilhyeon, thanks for your great work! Now I'm following your BMUE and have some trouble in reproducing the results on ActivityNet dataset.

I have tried to do some experiments on ActivityNet v1.2. I downloaded the I3D features provided by this link and adapt them to BMUE format. The following are some of my results. All experiments run for 6k epochs, the results are showed in the form of "(average_mAP, Test_acc)":

  • [no params changed] (0.0390, 0.43)

According to Sec 4.1 in your arXiv paper, T=50, so I set num_segments as 50 and do the following experiments:

  • [num_segments: 50] (0.0275, 0.31)

    • [class_th: 0.1] (0.0332, 0.36)
    • [class_th: 0.1, act_thresh_cas: np.arange(0.0, 0.15, 0.015)] (0.0400, 0.36)

Besides, I also tried to change "act_thresh_magnitudes", "NMS thresh", "alpha", "_lambda & gamma in get_proposal_oic()", etc. The results don't seem to get better: the test accuracy is around 0.4 and the average_mAP is very low. It's hard for me to find the best settings. Could you please share your params settings on ActivityNet 1.2 & 1.3 datasets? Or give me some advice on which params to change?

Looking forward to your reply and I'd be glad to cite your excellent work. Thanks!

hi @tobbysong, have you got the better result after you set the hyperparameters as mentioned?

kiwiJ commented 3 years ago

@xumh-9 @cecilia930426 @sauradip ActivityNet1.2 from W-TALC github. ActivityNet1.3 from A2CL-PT github.

hawklucky commented 3 years ago

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful. @Pilhyeon

liming-ai commented 3 years ago

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

hawklucky commented 3 years ago

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

liming-ai commented 3 years ago

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

The best result I got is 40.2, I sent email to author, but he cannot provide any training log and told me the random seed is unknown for the best result...I have tried many random seed but it did not work...

Richard-61 commented 2 years ago

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

The best result I got is 40.2, I sent email to author, but he cannot provide any training log and told me the random seed is unknown for the best result...I have tried many random seed but it did not work...

Me too. I run the code and only get the result of 40.2 mAP about THUMOS14.

Jockey721 commented 2 years ago

how about the settings of 'class_th', 'act_thresh_cas', 'act_thresh_magnitudes'? Could you provide a complete configure file for ActivityNet1.2 and ActivityNet1.3? That will very helpful.@kiwiJ

Hi, could you reproduce the result of 41.8 mAP about THUMOS14 in paper without their pre-trained model?

Unfortunately, it's only around 40.0 mAP

The best result I got is 40.2, I sent email to author, but he cannot provide any training log and told me the random seed is unknown for the best result...I have tried many random seed but it did not work...

Me too. I run the code and only get the result of 40.2 mAP about THUMOS14.

Me too. Has anyone reproduced the reported performance?