anet_qa acc is 0 - Githubissues

jayleicn / singularity

[ACL 2023] Official PyTorch code for Singularity model in "Revealing Single Frame Bias for Video-and-Language Learning"

https://arxiv.org/abs/2206.03428

MIT License

129 stars 13 forks source link

anet_qa acc is 0 #29

Closed fake-warrior8 closed 1 year ago

fake-warrior8 commented 1 year ago

Hi, I reran the eval_vqa.sh on anet dataset, however, I got an acc 0. I compressed the anet video used your script and ran the eval_vqa.sh as follows

python -m torch.distributed.run --nproc_per_node=1 --master_port 29503  tasks/vqa.py \
  ./configs/qa_anet.yaml \
  output_dir=${output_dir} \
  pretrained_path=${pretrained_path} \
  evaluate=True

The outputs are

[32m2023-06-01T18:11:40 | utils.basic_utils: [0m[evaluation] Generating answers: Total time: 0:01:48 (0.3850 s / it) [32m2023-06-01T18:11:40 | dataset.utils: [0mresult file saved to ./outputs/debug/val_latest.json [32m2023-06-01T18:11:40 | tasks.vqa_utils: [0mMissing predictions for 0 questions, example[:3] [] [32m2023-06-01T18:11:40 | main: [0mSkip eval test split. All test_types ['val'] [32m2023-06-01T18:11:40 | main: [0meval_res {'val': {'overall': 0.0}} [32m2023-06-01T18:11:40 | main: [0mTraining time 0:01:48 [32m2023-06-01T18:11:40 | main: [0mbest epoch 0 [32m2023-06-01T18:11:40 | main: [0mCheckpoints and Logs saved at ./outputs/debug

And all predictions are 'rim's. {"questionid": "v-uR5-jYe0Ag_3", "answer": "rim"}, {"question_id": "v_dJ0kxnyVzFI_2", "answer": "rim"}, {"question_id": "v_ekbZecn088U_9", "answer": "rim"}