search

Seth-Park / RobustChangeCaptioning

Code and dataset release for Park et al., Robust Change Captioning (ICCV 2019)
47 stars 11 forks source link

The pre-trained model provided in this repository different from the model used in the paper? #3

Open doiken23 opened 3 years ago

doiken23 commented 3 years ago

Using the pre-trained model provided in this repository, I ran the following command to calculate the performance of the model on the test data.

python test.py --cfg configs/dynamic/dynamic.yaml --visualize --snapshot 9000 --gpu 0

python evaluate.py \
    --results_dir ./experiments/dynamic/test_output/captions \
    --anno ./data/total_change_captions_reformat.json \
    --type_file ./data/type_mapping.json

The calculated scores are as follows:

BLEU-4 CIDEr METEOR SPICE
0.5355 1.1496 0.3794 0.3128

This score is slightly higher than the score provided in the paper. Is the model provided in this repository different from the model used in the paper?

Ph0eNiX0803 commented 3 years ago

Hi! @doiken23 Did you reproduce the score by training a new model using the code provided in this repository?

Ph0eNiX0803 commented 3 years ago

I train a model that score can't reach the pre-trained model.

doiken23 commented 3 years ago

@Ph0eNiX0803

Sorry, I didn't reproduce the score in the paper... Maybe the seed is not fixed, so the score will vary.