How to finetune with my own dataset?

[thiner]

Thanks for your great work and kindly open source to the public community! I am wondering how to finetune(or maybe other correct terms) this model with my own dataset? I read the training description, but I still have questions:

1. Should I repeat the two stages if I want to tune the model? Can I jump into stage 2 directly with the checkpoint file you provided and my dataset?
2. Need I append my own dataset to dataset folder provided at stage 2, or just replace it?

Thanks in advance!

[eeeuu]

dataset/README_2_STAGE.md

[thiner]

dataset/README_2_STAGE.md

I read the document, and then I raised the two questions. Maybe it's quite clear to you, but could you answer my questions or show me anexample how to tune the model with your own dataset?

[TsuTikgiau]

Hello! Our stage 2 focuses on teaching the model to talk in a natural way. If your own dataset can also serve for this purpose, you can ignore our stage2 dataset. If your dataset focuses on improving the model's ability to see something, I guess you can insert an additional training stage using your own dataset between stage 1 and stage 2.

[thiner]

Hello! Our stage 2 focuses on teaching the model to talk in a natural way. If your own dataset can also serve for this purpose, you can ignore our stage2 dataset. If your dataset focuses on improving the model's ability to see something, I guess you can insert an additional training stage using your own dataset between stage 1 and stage 2.

Thanks a lot for your guidance! Regarding the middle step between stage 1 and stage 2, I guess I should follow the stage 2 manual to setup, am I right?

[yPanStupidog]

Could you successfully fine-tune the 7B model? I met problem about padding.

[TsuTikgiau]

@thiner Our original stage1 and stage2 use the same training script train.py. The difference is in the config file. In your case, you can create a new training config file based on stage2's config file, change the dataset in the config, modify the hyperparameter, etc. Also, when you want to add a new dataset, you might want to check how the datasets in this repository are built. Check the folder minigpt4/datasets to get some ideas. Thanks!

[thiner]

@TsuTikgiau Thanks a lot, really helpful!

[thiner]

Could you successfully fine-tune the 7B model? I met problem about padding.

Yes, I ran the fine tuning stage(stage 2) sucessfully, but the result is not good till now, I am still trying to make it work. Regarding "padding issue", I believe there are many relevant issues raised by other users, you may search for solution.

[Zenobia7]

@thiner Do you see a way to evaluate the model? I've been poking around the code for a long time without looking at the code and introduction to model evaluation and validation data sets

[cnxupupup]

same question

[SLKAlgs]

@TsuTikgiau hello！May I know where is the evaluation dataset "cc_sbu_align" for stage2? I encountered this prompt when running stage2: "No validation splits found." How is the generated text validated? Additionally, I would like to fine-tune stage2 with my own dataset. Do I only need to replace "MiniGPT-4/train_configs/minigpt4_stage2_finetune.yaml" with my own dataset in the "minigpt4_stage2_finetune.yaml" file to use it?

[SLKAlgs]

@thiner hello！May I know where is the evaluation dataset "cc_sbu_align" for stage2? I encountered this prompt when running stage2: "No validation splits found." How is the generated text validated? Additionally, I would like to fine-tune stage2 with my own dataset. Do I only need to replace "MiniGPT-4/train_configs/minigpt4_stage2_finetune.yaml" with my own dataset in the "minigpt4_stage2_finetune.yaml" file to use it?

[zhangwengyu999]

Same question. How to evalute the the finetuning process? I set evaluate: True in the minigpt4_stage2_finetune.yaml, but it says No validation splits found.. Where and how should I put my validation splits? Thank you.

[Davidwhw]

Same question. How to evalute the the finetuning process? I set evaluate: True in the minigpt4_stage2_finetune.yaml, but it says No validation splits found.. Where and how should I put my validation splits? Thank you.

The verification step between training multiple epochs does not appear to be implemented in minigpt's code (such as lavis valid_step). Have you found a solution to your problem?