Fine-tuning InstructBLIP?

[iamwangyabin]

It seems that their fine-tuning strategy is similar to the standard training approach for VQA. I noticed that the blip2_vicuna_instruct.py file includes a predict_answers function, which is commonly used in VQA tasks.

To use their approach, you can prepare your datasets as they've described, including image, text_input, and text_output, and then launch train.py. However, I would also like to see more training details to better understand their methodology.

[chloejiang]

It seems the run_scripts do not include anything related to InstuctBlip. Will the official release the code for pre-training and fine-tuning of InstructBlip?

[austinmw]

Bump, could you expand on the Instruction-Tuning section for the InstructBLIP model page? It's not clear how to do this or even if the full code necessary has been released.

[edchengg]

Could you release VQA finetuning script for instructBLIP?

[aopolin-lv]

Could you release VQA finetuning script for instructBLIP?

meet the same problem

[hangzeli08]

same question

[qwqwq1445]

same question

[tigerzjh]

same question,how to fine tune instruct blip?

[floriankark]

I would also like to know how to fine tune instruct blip

[Richar-Du]

Could the author provide the fine-tuning script of InstructBLIP? @LiJunnan1992

[Tower0823]

mark!

[dydxdt]

mark

[g2zr004]

mark

[Oklahomawhore]

mark!

[sdc17]

same question here

[lxmcwt]

mark

[gwyong]

same question

[control-spiderman]

+1

[liu3xing3long]

+1, pretrain vicuna at stage2 also need to be modified

[Lanyu0303]

mark

[him-mah10]

+1

[santaboi]

mark

[Clement25]

mark

[Yuancheng-Xu]

Guys, are there any training scripts for InstructBLIP on captioning (not VQA) tasks? Something like https://github.com/salesforce/LAVIS/blob/main/run_scripts/blip2/train/train_caption_coco.sh but for InstructBLIP?

[owlsan49]

mark

[idor980]

mark

[dszpr]

mark！

[findalexli]

https://github.com/AttentionX/InstructBLIP_PEFT?tab=readme-ov-file It semms like the authors have no motivatioon to release the finetuning script, but here is a repo that claims to do the same

[waitzkin]

We have released the finetuning scripts, so let me know if you have any problem!

[Clement25]

We have released the finetuning scripts, so let me know if you have any problem!

Could you please tell me where it is? I found no finetuning scripts for instructblip under run_scripts/train/BLIP2

[waitzkin]

As in Train > Run Script section in README, it's under run_scripts/instructblip/train.

[Clement25]

As in Train > Run Script section in README, it's under run_scripts/instructblip/train.

Hi waitzkin, thanks for your reply. Could you please tell me which branch you are on? On the main branch there's no folder named 'instructblip' in run_scripts folder.

[waitzkin]

Main branch is the correct branch, and the script is here. https://github.com/AttentionX/InstructBLIP_PEFT/tree/main/run_scripts/instructblip/train The instructblip directory only has one sub directory 'train', so it is denoted as instructblip/train here(https://github.com/AttentionX/InstructBLIP_PEFT/tree/main/run_scripts), and this might have confused you.

[lsnls]

mark