Understanding Garbled Text in vocab.json of Multimodal Models: Causes and Explanations

QwenLM / Qwen2-VL

Qwen2-VL is the multimodal large language model series developed by Qwen team, Alibaba Cloud.

Apache License 2.0

2.42k stars 133 forks source link

Understanding Garbled Text in vocab.json of Multimodal Models: Causes and Explanations #160

Open xddun opened 3 weeks ago

xddun commented 3 weeks ago

I noticed that the vocab.json file in the multimodal model appears to contain garbled text, for example:

"Ġnegot": 11642,
"Ġpointers": 27454,
"çŃĸçķ¥": 104238,
"Ġdelights": 95675,
"ĠArrayAdapter": 49640,
"Ġfv": 61654,
"åĲĳç¤¾ä¼ļ": 111834,
"nowledge": 51186,
"çļĦä»»åĬ¡": 108530,
"ĠØ§ÙĦØ·ÙĦØ§Ø¨": 140123,
"Ġacc": 1029,
".crt": 93869,
"(save": 33546,

Could someone explain the reason behind this garbled text and the underlying principles?

xddun commented 3 weeks ago

https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct/tree/main