search

XiaoduoAILab / XmodelVLM

Apache License 2.0
58 stars 2 forks source link

TypeError: not a string #1

Closed harrytrolor closed 5 months ago

harrytrolor commented 5 months ago

hi,I have the problem when I tried to run the inference.py ? It seems that the vocab_file load is wrong ,how can I fix it? by the way, I see the parameters VOCAB_FILES_NAMES = {"vocab_file": "xmodel_32000.model"},what does xmodel_32000 mean?because I didnot find it in the code, making me a bit confused

Errors about the first question:

Traceback (most recent call last):
  File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/inference.py", line 84, in <module>
    inference_once(args)
  File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/inference.py", line 22, in inference_once
    tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.load_8bit, args.load_4bit)
  File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/xmodelvlm/model/xmodelvlm.py", line 328, in load_pretrained_model
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=False)
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 801, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2029, in from_pretrained
    return cls._from_pretrained(
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2261, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/hubble/.cache/huggingface/modules/transformers_modules/Xmodel_VLM/tokenization_xmodel.py", line 76, in __init__
    self.sp_model.Load(vocab_file)
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string
(xmodelvlm) hubble@hubble-zhang:/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main$ python inference.py load_in_4bit
Traceback (most recent call last):
  File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/inference.py", line 84, in <module>
    inference_once(args)
  File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/inference.py", line 22, in inference_once
    tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.load_8bit, args.load_4bit)
  File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/xmodelvlm/model/xmodelvlm.py", line 328, in load_pretrained_model
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=False)
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 801, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2029, in from_pretrained
    return cls._from_pretrained(
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2261, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/hubble/.cache/huggingface/modules/transformers_modules/Xmodel_VLM/tokenization_xmodel.py", line 76, in __init__
    self.sp_model.Load(vocab_file)
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
  File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string
New-Star-HLP commented 5 months ago

This was an oversight on our part and the 'xmodel_32000.model' has already been uploaded on huggingface

harrytrolor commented 5 months ago

@New-Star-HLP Thanks a lot , I continued to run the inference.py and I changed the clip to siglip ,And I got another problem,I thought It was the different structure between them at first, but it still raised the same error after I changed Clip back.How should I fix it. By the way , what should I do to ues siglip or I just have to change the name and path?

Traceback (most recent call last): File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/inference.py", line 83, in <module> inference_once(args) File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/inference.py", line 21, in inference_once tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.load_8bit, args.load_4bit) File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/xmodelvlm/model/xmodelvlm.py", line 330, in load_pretrained_model model = XmodelLMForCausalLM.from_pretrained(model_path, trust_remote_code=True, low_cpu_mem_usage=True, **kwargs) File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3850, in from_pretrained ) = cls._load_pretrained_model( File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4284, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/modeling_utils.py", line 805, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 345, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([2048, 1024]) in "weight" (which has shape torch.Size([2048, 4096])), this look incorrect.

harrytrolor commented 5 months ago

@New-Star-HLP Thanks. But it seems that some other components remain incorrect after I update the vision_projector.py . Still the same error but the detailed shape size has changed

Traceback (most recent call last): File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/inference.py", line 83, in <module> inference_once(args) File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/inference.py", line 21, in inference_once tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.load_8bit, args.load_4bit) File "/media/hubble/E01EEDCB1EED9B34/XmodelVLM-main/xmodelvlm/model/xmodelvlm.py", line 330, in load_pretrained_model model = XmodelLMForCausalLM.from_pretrained(model_path, trust_remote_code=True, low_cpu_mem_usage=True, **kwargs) File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3850, in from_pretrained ) = cls._load_pretrained_model( File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4284, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/transformers/modeling_utils.py", line 805, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File "/home/hubble/anaconda3/envs/xmodelvlm/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 345, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([257, 1024])), this look incorrect.

New-Star-HLP commented 5 months ago

Thanks for the feedback, I suggest you update vision_projector.py and huggingface. Finally I have verified that there should be no errors. 17dbd27f-3c6a-4ed6-add0-a5ce2396bc5c