search

microsoft / unilm

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
https://aka.ms/GeneralAI
MIT License
20.1k stars 2.55k forks source link

Problems when fine-tune BEiT3 on VQA #1099

Open Alchemistyui opened 1 year ago

Alchemistyui commented 1 year ago

Thank you for your excellent work!

I cannot reproduce your result when I'm fine-tuning BEiT3 following your guidance get_started_for_vqav2.md. I'd appreciate it if you can give me some suggestions.

Describe I'm using 8 40G-A100 for fine-tuning, and not using docker but using conda environment. The other setups are the same as the guidance.

I've tried to train BEiT-3 base and BEiT-3 large with the script you provided, but there is still a big gap between my implementation performance and yours, as shown in the table:

Dataset Model Official Result My result
VQAv2 val. beit3_base 86.1% 74.40%
VQAv2 test-dev beit3_base 77.65% 73.1%
VQAv2 test-dev beit3_large 81.85% 68.8%

The train loss and accuracy looks fine (as follows), and I also upload the log, conda list and tensorboard file here: beit3 finetune log.

tensorboard curve

I'd appreciate it if you can help me. Thank you very much!

OrangeNo42 commented 1 year ago

May I ask how did you download beit-v3 pre-trained weights? I got PublicAccessNotPermitted error when trying to download beit-v3 pre-trained weights from https://conversationhub.blob.core.windows.net/beit-share-public/beit3/vqa/beit3_base_patch16_480_vqa.pth

` PublicAccessNotPermitted

Public access is not permitted on this storage account. RequestId:c41c88a3-901e-0059-75cf-997f8e000000 Time:2023-06-08T06:06:09.4120160Z

`

Kizna1ver commented 1 year ago

May I ask how did you download beit-v3 pre-trained weights? I got PublicAccessNotPermitted error when trying to download beit-v3 pre-trained weights from https://conversationhub.blob.core.windows.net/beit-share-public/beit3/vqa/beit3_base_patch16_480_vqa.pth

<Error> <Code>PublicAccessNotPermitted</Code> <Message>Public access is not permitted on this storage account. RequestId:c41c88a3-901e-0059-75cf-997f8e000000 Time:2023-06-08T06:06:09.4120160Z</Message> </Error>

I meet the same problem when downloading initialized checkpoint of COCO Retrieval.

Alchemistyui commented 1 year ago

May I ask how did you download beit-v3 pre-trained weights? I got PublicAccessNotPermitted error when trying to download beit-v3 pre-trained weights from https://conversationhub.blob.core.windows.net/beit-share-public/beit3/vqa/beit3_base_patch16_480_vqa.pth

<Error> <Code>PublicAccessNotPermitted</Code> <Message>Public access is not permitted on this storage account. RequestId:c41c88a3-901e-0059-75cf-997f8e000000 Time:2023-06-08T06:06:09.4120160Z</Message> </Error>

Alchemistyui commented 1 year ago

May I ask how did you download beit-v3 pre-trained weights? I got PublicAccessNotPermitted error when trying to download beit-v3 pre-trained weights from https://conversationhub.blob.core.windows.net/beit-share-public/beit3/vqa/beit3_base_patch16_480_vqa.pth

<Error> <Code>PublicAccessNotPermitted</Code> <Message>Public access is not permitted on this storage account. RequestId:c41c88a3-901e-0059-75cf-997f8e000000 Time:2023-06-08T06:06:09.4120160Z</Message> </Error>

I download the pretrained weight according to their table here: https://github.com/microsoft/unilm/tree/master/beit3#fine-tuning-on-vqav2-visual-question-answering.

For example, beit3_base_patch16_224.