About the best result

[Hugh1222]

I would like to ask a question about the result. I have been attempting to replicate the best results mentioned in your paper using the provided train_yc2.sh. However, I seem to be unable to achieve the same outcomes.

Could you kindly confirm if there are any additional parameters or modifications that I might need to adjust in the script to attain the best results as reported? Any guidance or suggestions you could provide would be greatly appreciated.

The results from one of the trials are as follows. {'Bleu_1': 0.12333597612177397, 'Bleu_2': 0.06884653306042646, 'Bleu_3': 0.03383332827651567, 'Bleu_4': 0.013209217421733404, 'METEOR': 0.05714311522225237, 'ROUGE_L': 0.11398654315764384, 'CIDEr': 0.28838498331546225, 'Recall': 0.24384769314528523, 'Precision': 0.3258541757681666, 'soda_c': 0.05066747655389695, 'para_Bleu_1': 0.38141937171062135, 'para_Bleu_2': 0.2285028131410581, 'para_Bleu_3': 0.13738682567006053, 'para_Bleu_4': 0.08491109673372983, 'para_METEOR': 0.1468747734479021, 'para_ROUGE_L': 0.2949792652901276, 'para_CIDEr': 0.14848706801072795, 'MetaScore': 0.10781059177614932, 'grounding_mIOU': 0.0, 'grounding_R@1IOU0.7': 0.0, 'grounding_R@5IOU0.7': 0.0, 'grounding_R@1IOU0.5': 0.0, 'grounding_R@5IOU0.5': 0.0, 'grounding_R@1IOU0.3': 0.0, 'grounding_R@5IOU0.3': 0.0, 'grounding_R@1IOU0.1': 0.0, 'grounding_R@5IOU0.1': 0.0, 'aux_grounding_mIOU': 0.0, 'aux_grounding_R@1IOU0.7': 0.0, 'aux_grounding_R@5IOU0.7': 0.0, 'aux_grounding_R@1IOU0.5': 0.0, 'aux_grounding_R@5IOU0.5': 0.0, 'aux_grounding_R@1IOU0.3': 0.0, 'aux_grounding_R@5IOU0.3': 0.0, 'aux_grounding_R@1IOU0.1': 0.0, 'aux_grounding_R@5IOU0.1': 0.0}

[tuyunbin]

How many gpus do you use?

[tuyunbin]

Here are my re-implemented results without any modification 'Bleu_1': 0.12994438568313482, 'Bleu_2': 0.07225012066639824, 'Bleu_3': 0.03487183135604932, 'Bleu_4': 0.014392946606256021, 'METEOR': 0.05843631592069194, 'ROUGE_L': 0.11802471347488694, 'CIDEr': 0.3075642898881613, 'Recall': 0.24718669932259407, 'Precision': 0.3266842325615263, 'soda_c': 0.052156796274662964,

[tuyunbin]

I run this code on yc2 dataset several times, and find that the results are different. I guess that the seed is not successfully fixed. I wonder about whether you meet the same problem? @Hugh1222

[Hugh1222]

I only used one GPU. Even though I fixed the random seed that can be set in the parameters, I still encountered the issue of differing results.Have the results you obtained from running this code multiple times matched those reported in the paper? @tuyunbin

[tuyunbin]

Hi, I find that the seed cannot be fixed due to the use of this function: F.grid_sample in "/CM2_DVC/cm2/ops/functions/ms_deform_attn_func.py". I have run several times and found that some metrics' results are close to the paper.

[Hugh1222]

I noticed that most of the results you provided above still not close to those reported in the paper. May I ask if, apart from the seed's influence, you made any other changes in subsequent attempts?

[tuyunbin]

In my opinion, the seed's influence is non-negligible, because I will not be sure whether good results are obtained due to my attempt or just the random results.

[ydwl-lynn]

In my opinion, the seed's influence is non-negligible, because I will not be sure whether good results are obtained due to my attempt or just the random results.

Hi, when I was re-training the activitynet dataset, I found that I couldn't save the parameter file (.pth) for each epoch, and the address of the 'gt_file_for_eval' file was always reporting an error when testing, showing: gt_file_for_eval = [ 'data/anet/captiondata/val_1.json', 'data/anet/captiondata/val_2.json']. Have you encountered this situation, please? How to solve it?

[ydwl-lynn]

I noticed that most of the results you provided above still not close to those reported in the paper. May I ask if, apart from the seed's influence, you made any other changes in subsequent attempts?

Hi, when I was re-training the activitynet dataset, I found that I couldn't save the parameter file (.pth) for each epoch, and the address of the 'gt_file_for_eval' file was always reporting an error when testing, showing: gt_file_for_eval = [ 'data/anet/captiondata/val_1.json', 'data/anet/captiondata/val_2.json']. Have you encountered this situation, please? How to solve it?