The accuracy of model test is poor

[aoji0606]

Thanks for your work.

I use the siglip and llama3 for pretrain and fine-tune，but the test results were poor，here are the results： Why is that?

[Isaachhh]

We use a better strategy to train Phi-3-Mini-based and Llama-3-8B-based Bunny, which would be open-sourced soon.

BTW, do you use Bunny-pretrain-laion-2m and Bunny-695k? It seems that your results are a little bit worse than expectation.

[aoji0606]

Yes, I use Bunny-pretrain-laion-2m for pre-train, and Bunny-695k for fine-tune

[Isaachhh]

What's your pre-training learning rate?

[aoji0606]

1e-3 for pre-train 2e-5 for fine-tune

[Isaachhh]

Do you use lora or full fine-tuning?

[aoji0606]

I use full fine-tuning

[Isaachhh]

LoRA would convey a better performance.

[aoji0606]

ahhh, ok, thank you

[aoji0606]

hi~ What the learning rate should be set when using lora to fine-tune the llama3 model

[Isaachhh]

As default, 2e-4 and 2e-5.

[aoji0606]

ok thank u

[Isaachhh]

Close the issue for now if there's no further discussions. Feel free to reopen it if there's any other questions.