search

sLdZ0306 / CCR

Counterfactual Cross-modality Reasoning for Weakly Supervised Video Moment Localization
MIT License
8 stars 0 forks source link

Problems on reproducing the results in the paper on Charades #1

Open akzeycgdn265 opened 1 year ago

akzeycgdn265 commented 1 year ago

Hello! Thanks for your outstanding work!

When I tried to reproduce your paper results and run your paper code, the results on the charades dataset were much lower than the results in your paper, and not even higher than the results of your baseline model CPL under the same circumstances.

Here are the results of my run: CCR results：
| R@1,mIoU 0.4334 | R@1,IoU@0.3 0.6697 | R@1,IoU@0.5 0.4797 | R@1,IoU@0.7 0.2220 | R@5,mIoU 0.6771 | R@5,IoU@0.3 0.9699 | R@5,IoU@0.5 0.8439 | R@5,IoU@0.7 0.5076 CPL results： | R@1,mIoU 0.4345 | R@1,IoU@0.3 0.6650 | R@1,IoU@0.5 0.4902 | R@1,IoU@0.7 0.2277 | R@5,mIoU 0.6772 | R@5,IoU@0.3 0.9671 | R@5,IoU@0.5 0.8452 | R@5,IoU@0.7 0.5155

Can you tell me the specific version number of your code running environment? Do I need to change any hyperparameters when I run the code?

Looking forward to your reply!

sLdZ0306 commented 1 year ago

Thank you for your interest!

To reproduce the results in Charades-STA dataset, please include '--vote' in the training command to utilize a loss-based strategy for selecting the optimal candidate proposals instead of the vote-based strategy. Further details regarding this selection strategy can be found in our paper.

I hope this is helpful to you.

akzeycgdn265 commented 1 year ago

Thank you for your interest!

To reproduce the results in Charades-STA dataset, please include '--vote' in the training command to utilize a loss-based strategy for selecting the optimal candidate proposals instead of the vote-based strategy. Further details regarding this selection strategy can be found in our paper.

I hope this is helpful to you.

Thank you for your reply! I tried to use the '--vote' command today, but there is still a big gap with your paper results on some metrics in the charades dataset and the activity dataset.

Here are the results of my run: CharadesSTA-not vote | R@1,mIoU 0.4334 | R@1,IoU@0.3 0.6697 | R@1,IoU@0.5 0.4797 | R@1,IoU@0.7 0.2220 | R@5,mIoU 0.6771 | R@5,IoU@0.3 0.9699 | R@5,IoU@0.5 0.8439 | R@5,IoU@0.7 0.5076

CharadesSTA-vote | R@1,mIoU 0.4346 | R@1,IoU@0.3 0.6719 | R@1,IoU@0.5 0.4816 | R@1,IoU@0.7 0.2226 | R@5,mIoU 0.6771 | R@5,IoU@0.3 0.9699 | R@5,IoU@0.5 0.8439 | R@5,IoU@0.7 0.5076

activitynet-not vote | R@1,mIoU 0.3320 | R@1,IoU@0.3 0.4778 | R@1,IoU@0.5 0.2541 | R@5,mIoU 0.3968 | R@5,IoU@0.3 0.5619 | R@5,IoU@0.5 0.3374

activitynet-vote | R@1,mIoU 0.3474 | R@1,IoU@0.1 0.8204 | R@1,IoU@0.3 0.4863 | R@1,IoU@0.5 0.2217 | R@5,mIoU 0.4447 | R@5,IoU@0.1 0.8812 | R@5,IoU@0.3 0.6190 | R@5,IoU@0.5 0.4127

Can you publish the specific environment version of your program? Also, are the code hyperparameters you publish the same as the hyperparameters you actually use?

Looking forward to your reply!

sLdZ0306 commented 1 year ago

Hi! The evironment are detailed in the requirements.txt file, and the setting of hyperparameters should also be fine. However, I've noticed that the current version of this repository is slightly inconsistent with the paper version, and I will update this when I'm free. Sorry for the inconvenience.

akzeycgdn265 commented 12 months ago

Hi! The evironment are detailed in the requirements.txt file, and the setting of hyperparameters should also be fine. However, I've noticed that the current version of this repository is slightly inconsistent with the paper version, and I will update this when I'm free. Sorry for the inconvenience.

Thank you very much for your patient reply! I have done the experiment again in accordance with the environment you published these days, and I can reproduce the results similar to your paper on the Charades dataset, but the results of R5 in ActivityNet dataset are much lower than the results of your paper. I notice that your baseline model CPL is very sensitive to the alpha_2 parameter on the ActivityNet dataset. Did you use 1 for parameter alpha_2 in ActivityNet's R5 experiment?

Here are the results of my ActivityNet data set running: "alpha_2": 0.1 | R@1,mIoU 0.3645 | R@1,IoU@0.1 0.7973 | R@1,IoU@0.3 0.5329 | R@1,IoU@0.5 0.3057 | R@5,mIoU 0.4779 | R@5,IoU@0.1 0.8859 | R@5,IoU@0.3 0.6674 | R@5,IoU@0.5 0.4710

Looking forward to your reply!

sLdZ0306 commented 12 months ago

Hi! The hyperparameters should be the same for both R@1 and R@5. During the optimization process, the performances w.r.t. R@1 and R@5 often do not improve synchronously. However, checkpoints are selected based on the R@1 mIoU metric. So please check if there exists other checkpoints with more balanced performance on different metrics. Finally, please confirm the voting strategy is used on ActivityNet.

akzeycgdn265 commented 12 months ago

Hi! The hyperparameters should be the same for both R@1 and R@5. During the optimization process, the performances w.r.t. R@1 and R@5 often do not improve synchronously. However, checkpoints are selected based on the R@1 mIoU metric. So please check if there exists other checkpoints with more balanced performance on different metrics. Finally, please confirm the voting strategy is used on ActivityNet.

Hello! I conducted many experiments on ActivityNet dataset according to your suggestion, but unfortunately the results on R5 are still far from those in your paper. Can you publish your trained model weights like CPL?

sLdZ0306 commented 12 months ago

I will check and update this repo when I'm free in the future.

akzeycgdn265 commented 11 months ago

I will check and update this repo when I'm free in the future.

Hello! When will you update your code repository? Looking forward to your update!