how can I replace tokenizer TinyChart

[LilDevsy0117]

I want to change the tokenizer so that it can be applied to Korean

I would appreciate it if you could change LLM_PATH and additionally let me know which part of the code should be modified.

[zhangliang-04]

Hi @LilDevsy0117, Our model is based on TinyLlava, which uses Phi-2 as the LLM. It uses Byte-level BPE and can be naturally applied to other languages including Korean after fine-tuning your data. However, we cannot guarantee its performance, which is related to the Korean support of the Phi-2. If you want to use another LLM, try to write a class like llava_phi that support your LLM to receive image features from ViT. Note that you may pre-train the projector and LLM on the general image-text datasets to get better performance just like TinyLlava does.

[LilDevsy0117]

Thanks @zhangliang-04 I am trying as below.

train.sh LLM_PATH=tabtoyou/KoLLaVA-v1.5-Synatra-7b # replaced VIT_PATH=mPLUG/TinyChart-3B-768-siglip

and write llava_synatra.py

modified train.py

config = LlavaConfig.from_pretrained(model_args.model_name_or_path) 86 model = LlavaLlamaForCausalLM.from_pretrained( 87 model_args.model_name_or_path, 88 config = config, 89 cache_dir=training_args.cache_dir, 90 **bnb_model_from_pretrained_args, 91 attn_implementation=None, 92 torch_dtype=compute_dtype 93 )

Tokenizer, init_tokenizer = TokenizerSelect('synatra')() 118 tokenizer = Tokenizer.from_pretrained( 119 model_args.model_name_or_path, 120 cache_dir=training_args.cache_dir, 121 model_max_length=training_args.model_max_length, 122 padding_side="right", 123 use_fast=True, 124 ) 125 tokenizer = init_tokenizer(tokenizer)

However, I encountered the following error.

You are using a model of type llava to instantiate a model of type tiny_chart_synatra. This is not supported for all configurations of models and can yield errors. WARNING:tinychart.model.multimodal_encoder.siglip_encoder:You are using a model of type clip to instantiate a model of type siglip_vision_model. This is not supported for all configurations of models and can yield errors.

WARNING: tokenization mismatch: 203 vs. 210. (ignored) number of rounds: 1 rounds: ["A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: \nGenerate underlying data table of the chart. ASSISTANT: TITLE | 기술 및 기능부문에서 장비 규격 유지관리의 평가방법은 무엇인가 \n | 일반부문 및 안개요(20점) | 장비 기능 요구사항 \n 0 | 40 | 18 \n 1 | 31 | 2 \n 2 | 8 | 4 \n 3 | 20 | 9 \n 4 | 14 | 34 \n 5 | 20 | 41 \n 6 | 20 | 7"] conversation: ["A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: \nGenerate underlying data table of the chart. ASSISTANT: TITLE | 기술 및 기능부문에 서 장비 규격 유지관리의 평가방법은 무엇인가 \n | 일반부문 및 안개요(20점) | 장비 기능 요구사항 \n 0 | 40 | 18 \n 1 | 31 | 2 \n 2 | 8 | 4 \n 3 | 20 | 9 \n 4 | 14 | 34 \n 5 | 20 | 41 \n 6 | 20 | 7<|endoftext|>"] tensor([ -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 320, 1153, 1180, 342, 28705, 29164, 239, 139, 163, 28705, 31262, 28705, 29164, 30364, 29775, 29710, 29148, 29305, 28705, 29747, 29859, 28705, 31982, 31110, 28705, 30127, 29161, 30224, 29288, 29187, 28705, 31523, 29135, 30240, 30979, 29538, 28705, 30449, 239, 154, 138, 29324, 29135, 28705, 13, 28705, 342, 28705, 29415, 30192, 29775, 29710, 28705, 31262, 28705, 30325, 29893, 29517, 28732, 28750, 28734, 30589, 28731, 342, 28705, 29747, 29859, 28705, 29164, 30364, 28705, 29517, 29779, 29315, 30968, 28705, 13, 28705, 28734, 342, 28705, 28781, 28734, 342, 28705, 28740, 28783, 28705, 13, 28705, 28740, 342, 28705, 28770, 28740, 342, 28705, 28750, 28705, 13, 28705, 28750, 342, 28705, 28783, 342, 28705, 28781, 28705, 13, 28705, 28770, 342, 28705, 28750, 28734, 342, 28705, 28774, 28705, 13, 28705, 28781, 342, 28705, 28740, 28781, 342, 28705, 28770, 28781, 28705, 13, 28705, 28782, 342, 28705, 28750, 28734, 342, 28705, 28781, 28740, 28705, 13, 28705, 28784, 342, 28705, 28750, 28734, 342, 28705, 28787, 28789, 28766, 416, 1009, 772, 28766, 28767]) tensor([[ 1, 330, 10706, 1444, 264, 13903, 2188, 304, 396, 18278, 10895, 13892, 28723, 415, 13892, 5212, 10865, 28725, 10537, 28725, 304, 27057, 11194, 298, 272, 2188, 28742, 28713, 4224, 28723, 2223, 725, 28747, 28705, -200, 28705, 13, 23342, 14164, 1178, 2401, 302, 272, 10968, 28723, 8602, 8048, 12738, 28747, 320, 1153, 1180, 342, 28705, 29164, 239, 139, 163, 28705, 31262, 28705, 29164, 30364, 29775, 29710, 29148, 29305, 28705, 29747, 29859, 28705, 31982, 31110, 28705, 30127, 29161, 30224, 29288, 29187, 28705, 31523, 29135, 30240, 30979, 29538, 28705, 30449, 239, 154, 138, 29324, 29135, 28705, 13, 28705, 342, 28705, 29415, 30192, 29775, 29710, 28705, 31262, 28705, 30325, 29893, 29517, 28732, 28750, 28734, 30589, 28731, 342, 28705, 29747, 29859, 28705, 29164, 30364, 28705, 29517, 29779, 29315, 30968, 28705, 13, 28705, 28734, 342, 28705, 28781, 28734, 342, 28705, 28740, 28783, 28705, 13, 28705, 28740, 342, 28705, 28770, 28740, 342, 28705, 28750, 28705, 13, 28705, 28750, 342, 28705, 28783, 342, 28705, 28781, 28705, 13, 28705, 28770, 342, 28705, 28750, 28734, 342, 28705, 28774, 28705, 13, 28705, 28781, 342, 28705, 28740, 28781, 342, 28705, 28770, 28781, 28705, 13, 28705, 28782, 342, 28705, 28750, 28734, 342, 28705, 28781, 28740, 28705, 13, 28705, 28784, 342, 28705, 28750, 28734, 342, 28705, 28787, 28789, 28766, 416, 1009, 772, 28766, 28767]])