search

cambrian-mllm / cambrian

Cambrian-1 is a family of multimodal LLMs with a vision-centric design.
https://cambrian-mllm.github.io/
Apache License 2.0
1.77k stars 115 forks source link

Using Cambrian 7M as instruction data got extremly bad result #49

Open lucasjinreal opened 4 months ago

lucasjinreal commented 4 months ago

Normaly, I using my data got a reasonable result, but same training process, just replace data with Cambrian 7M, got some extremly bad result, why?

tsb0601 commented 4 months ago

Hi Could you please elaborate on it? Try using Cambrian7M with system prompts: https://huggingface.co/datasets/nyu-visionx/Cambrian-10M/resolve/main/jsons/Cambrian7M_withsystemprompt.jsonl

lucasjinreal commented 4 months ago

I tried using Cambiran 7M with systme prompts.

I don't know why, maybe the more longer training process, shouldn't use same wramup ratio？

I tried the intermediate model, the inference is not as good as expected, almost can be treat as over-fitted.

I am not sure how to dealing with so larger instruction data, any suggestions?

Haven't u guys met similar issue?

tsb0601 commented 4 months ago

Hi,

maybe try following our training hyperparameters? Also, we encountered similar issues but we later realized its bug with transformer version in the inference time. Maybe this will help a bit

lxtGH commented 3 months ago

@tsb0601 Hi! Could you please provide the detailed training hyperparameters or which file should I refer to?

lxtGH commented 2 months ago

Hello, I meet the same problem. Using this dataset results in inferior results than LLaVA. @lucasjinreal What are your setting?

wuziheng commented 2 weeks ago

same problem