Image Caption Evaluation

[IOEvan]

Thanks for releasing this great project. But, when I using the evaluation example script of COCO Captioning python run.py with data_root=/data2/dsets/dataset num_gpus=8 num_nodes=1 task_finetune_caption_mle_coco per_gpu_batchsize=32 load_path=fiber_coco_caption.ckpt

I get the following error:

Traceback (most recent calls WITHOUT Sacred internals):
  File "run.py", line 25, in main
    model = FIBERTransformerSS(_config)
  File "/homeu/project/OVOD/FIBER/coarse_grained/fiber/modules/fiber_module.py", line 146, in __init__
    state_dict = swin_adapt_position_encoding(
  File "/home/project/OVOD/FIBER/coarse_grained/fiber/modules/swin_helpers.py", line 32, in swin_adapt_position_encoding
    pos_embed = pos_embed.transpose(0, 1).view(-1, before, before)
RuntimeError: shape '[-1, 23, 23]' is invalid for input of size 4900

What is the problem that fiber_coco_caption.ckpt model does not match the input size? Hope to get your reply！

[zdou0830]

Thanks! Could you try changing 'task_finetune_caption_mle_coco' to 'task_finetune_caption_cider_coco' and see if it can fix the issue?

[IOEvan]

• The task_finetune_caption_cider_coco also does not work since I don't have the coco-train-words.p file

FileNotFoundError: [Errno 2] No such file or directory: 'coco-train-words.p'

• When I trying the task_finetune_caption_gold_coco, the following error arised

TypeError: __init__() got an unexpected keyword argument 'prepare_data_per_node'

Do you have any other suggestions?

[zdou0830]

coco-train-words.p can be found here https://drive.google.com/file/d/1N_O7tkBjJRCueQj7MHMqrazDTWr4OCDE/view?usp=sharing

TypeError: init() got an unexpected keyword argument 'prepare_data_per_node' seems like a pytorch-lightning version issue. here's the package version information https://github.com/microsoft/FIBER/blob/main/coarse_grained/requirements.txt.

if these don't work, maybe you can just try adding resolution_before=576 to the command line for task_finetune_caption_mle_coco

[IOEvan]

coco-train-words.p can be found here https://drive.google.com/file/d/1N_O7tkBjJRCueQj7MHMqrazDTWr4OCDE/view?usp=sharing

TypeError: init() got an unexpected keyword argument 'prepare_data_per_node' seems like a pytorch-lightning version issue. here's the package version information https://github.com/microsoft/FIBER/blob/main/coarse_grained/requirements.txt.

if these don't work, maybe you can just try adding resolution_before=576 to the command line for task_finetune_caption_mle_coco

---

Thanks for your advice! It works for me.