[CogVLM-chat-v1.1] LM weights are different with vicuna-7b-v1.5

[minostauros]

While CogVLM is trained, LM weights are fronzen.

From my observation however, the LM weights of cogvlm are different with Vicuna

Vicuna: https://huggingface.co/lmsys/vicuna-7b-v1.5/tree/main CogVLM: cogvlm-chat-v1.1 (both from HF or SAT)

Can I ask why or the proper source of the language model?

• CogVLM-Chat-v1.1 (SAT)

• Vicuna-7B-v1.5

[antigone660]

I have migrated the CogVLM-chat-hf to mindspore and found the modle works well when the input including image and text, but if there is only text query without image, the performance is not so good, it may relate to this issue I guess.

[minostauros]

I have migrated the CogVLM-chat-hf to mindspore and found the modle works well when the input including image and text, but if there is only text query without image, the performance is not so good, it may relate to this issue I guess.

@antigone660 In text-only mode, the prompt template is different. Did you use the following prompt for text-only query?

https://github.com/THUDM/CogVLM/blob/b37f36bf5df8c0356920b70f21338c1968c20e47/basic_demo/cli_demo_hf.py#L52

In my case, text-only mode works well regardless of this issue

[antigone660]

@minostauros Thanks for your reply I did not use the template before and it works now : )

[JiaQiSJTU]

So, the LM had also been tuned? I also found that the LM weights of cogvlm2 are different with llama-3-8B-Instruct.