some questions for train\val

[Mr-Monday]

Hi,

Thanks for the great work. I use the following commands to train/val the code ，but got the clip-classifier_val-mc.json dont have the key "multiple_choice" or "direct_answer". So when I use the "and use the "python evaluation/eval_predictions.py" for evaluation, ,I get bugs What kind of question is that? Figure 1 is a josn file generated using python transfer_experiments/predict.py. Figure 2 is a bug usingpython evaluation/eval_predictions.py.

In addition, when I use question and image as input, can I extract visual features from CLIP-ResNet50?

python transfer_experiments/train.py \ --aokvqa-dir ${AOKVQA_DIR} \ --vocab ${AOKVQA_DIR}/large_vocab_train.csv \ --log-dir ${LOG_DIR} \ --backbone clip \ --clip-model-type ViT-B/32 \ --train-features ${FEATURES_DIR}/clip-ViT-B-32_train.pt \ --val-features ${FEATURES_DIR}/clip-ViT-B-32_val.pt \ --inputs question image \ --objective classifier

python transfer_experiments/predict.py \ --aokvqa-dir ${AOKVQA_DIR} \ --out ${PREDS_DIR}/clip-classifier_val-mc.json \ --split val \ --features ${FEATURES_DIR}/clip-ViT-B-32_val.pt \ --ckpt logs/clip-classifier/inputs:question/checkpoints/epoch=494-val_acc=0.11.ckpt \ --inputs question image \ --mc \ --vocab ${AOKVQA_DIR}/large_vocab_train.csv

python evaluation/eval_predictions.py --aokvqa-dir ${AOKVQA_DIR} --split val --preds predictions/clip-classifier_val-mc.json

[apoorvkh]

Hey: before you run python evaluation/eval_predictions.py, I think you need to run

python evaluation/prepare_predictions.py --aokvqa-dir ${AOKVQA_DIR} --split val --mc ./predictions/clip-classifier_val-mc.json --out ./predictions/predictions_clip-classifier_val-mc.json

This merges the predictions from multiple choice and direct answer files into a format with {"multiple_choice": ..., "direct_answer": ...}. But the command above omits the --da (direct answer) argument, since you only have a multiple choice file.

And then you can run

python evaluation/eval_predictions.py --aokvqa-dir ${AOKVQA_DIR} --split val --preds ./predictions/predictions_clip-classifier_val-mc.json

Sorry for the awkward file naming.

[apoorvkh]

I'm pretty sure this should solve it, but please re-open if you still have issues!