I want to evaluate trained models using CIDER, etc.

[tnb1021]

I want to evaluate trained models using CIDER, but i can't it. When language_eval=0, it can be evaluated, but when language_eval=1, an error occurs.I want to use language_eval. tried:

python tools/eval.py --model log_fc/model-best.pth --infos_path log_fc/infos_fc-best.pkl --image_folder ../repurpose-gan/test_images/ --num_images -1 --force 1 --language_eval 1

then:

Traceback (most recent call last):
  File "/home/tanabe/ImageCaptioning.pytorch/tools/eval.py", line 79, in <module>
    lang_stats = eval_utils.language_eval(opt.input_json, predictions, n_predictions, vars(opt), opt.split)
  File "/home/tanabe/ImageCaptioning.pytorch/captioning/utils/eval_utils.py", line 84, in language_eval
    mean_perplexity = sum([_['perplexity'] for _ in preds_filt]) / len(preds_filt)
ZeroDivisionError: division by zero

This error has been reported in other issues, but all seem to be resolved by setting --force=1 and language_eval=0. I can eliminate the error with those conditions too, but what can I do to run with language_eval=1?

[ruotianluo]

It does not support evaluation on arbitrary image. You have to evaluate on coco or flickr30k, or you need to create your dataset in the same way as the coco format.

[tnb1021]

I prepared and ran the same format as coco and got the same error. Is there anything else needed other than the files in the following command?

python tools/eval.py --input_json data/dataset_cub_test.json --input_fc_dir data/cubtalk-test_fc/ --input_att_dir data/cubtalk-test_att/ --input_label_h5 none --num_images -1 --model log_fc/model-best.pth --infos_path log_fc/infos_fc-best.pkl --language_eval 1 --force 1

I use "cub" dataset.

[ruotianluo]

You need to prepare a Json file in coco-caption folder too.

[ruotianluo]

Similar to https://github.com/ruotianluo/ImageCaptioning.pytorch/tree/master/data#flickr30k.

You also need to add a line to let the coco-caption to read this Json file.