TopCoder2K
commented
3 years ago Good afternoon, @ashkamath!
Thank you for the great MDETR!
I've been trying to learn MDETR on the VQA2 dataset. As I wrote above, I managed to implement the code that is needed to run training on the VQA2. Then I decided to conduct your experiment from the Appendix E of the article. I fine-tuned on the GQA balanced with --no-detection option for 10 epochs and then fine-tuned on the VQA2 for 25 epochs. But the results are quite strange. It seems that the model hasn't learned, the loss on the GQA has increased and on the VQA almost hasn't changed. Evaluation during training was performed on the val and minival splits of the GQA and VQA2 respectively.
Here are the commands I've used. Fine-tuning on the GQA:
python run_with_submitit.py --dataset_config configs/gqa.json --ngpus 1 --nodes 1 --ema --epochs 10 --epoch_chunks 25 --do_qa --split_qa_heads --lr_drop 150 --resume https://zenodo.org/record/4721981/files/pretrained_resnet101_checkpoint.pth --batch_size 4 --no_detection --qa_loss_coef 25 --lr 1.4e-4 --lr_backbone 1.4e-5 --text_encoder_lr 7e-5 --output-dir ~/MDETR/mdetr/checkpointFine-tuning on the VQA2:
python run_with_submitit.py --dataset_config configs/vqa2.json --ngpus 1 --nodes 1 --epochs 25 --epoch_chunks 25 --do_qa --split_qa_heads --lr_drop 150 --backbone resnet101 --load ~/MDETR/mdetr/checkpoint/pchelintsev/experiments/19311/BEST_checkpoint.pth --batch_size 4 --no_aux_loss --no_contrastive_align_loss --no_detection --qa_loss_coef 25 --lr 1.4e-4 --lr_backbone 1.4e-5 --text_encoder_lr 7e-5 --output-dir ~/MDETR/mdetr/checkpoint --do_qa_with_qa_fine-tunedEvaluation on the VQA2:
python main.py --dataset_config configs/vqa2.json --eval --do_qa --split_qa_heads --no_contrastive_align_loss --no_aux_loss --no_detection --backbone resnet101 --qa_loss_coef 25 --resume ~/MDETR/mdetr/checkpoint/pchelintsev/experiments/5063/BEST_checkpoint.pthThe only significant difference is that I did 10 epochs on the GQA balanced while you did 5 epochs on the GQA all. But can it have such an impact? I think that hyperparameters can be wrong... Could you please provide what options and what hyperparameters values you used?
Also, it's interesting how you loaded the model from BEST_checkpoint.pth after fine-runing on the GQA? I used the --loadoption, and mismatching heads are deleted before running load_state_dict() (for example head for the types of questions).
VQA2, 10 epochs, torch.set_deterministic(True), after fine-tuning on GQA balanced for 5 epochs with --no_detection
ashkamath
Experiments with VQA v.2 dataset are described in Appendix E of the article. But it's not clear from main.py and run_with_submitit.py files how to run the fine-tuning (I've tried to write the same command that is used for fine-tuning on CLEVR). I've also found vqa_coco_format.py but it seems like preparation of the data, not fine-tuning itself. Also, I've encountered using build_dataset function in main.py and I don't see VQA v2 in the function :( Could you please explain how to do so?
UPD 1 (09.27.21): I've downloaded COCO and VQA v2 datasets and ran
And the processing has finished correctly. Now I'm thinking how to write VQA v2 dataset script...
UPD 2 (10.03.21) It seems I managed to implement all the necessary classes and fix the code. I'm currently doing an experiment eval -> train on vqa2 -> eval. As soon as it successfully finishes I'll push the code into my fork of the repo.
UPD 3 (10.03.21) Yeah, it works! Here is the link: https://github.com/TopCoder2K/mdetr. I haven't written any documentation because I'm not sure that fine-tuning on VQA is useful to anybody.)) If you have any question, please ask here :)