search

InternLM / InternLM-XComposer

InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
Apache License 2.0
2.5k stars 154 forks source link

Inference problem with my lora trained checkpoint #422

Open RBBB2010 opened 2 months ago

RBBB2010 commented 2 months ago

Firstly，thank you for the incredible results of this work.

I wrote the following inference code referring to the official document load the model after training, but it reported an error TypeError: 'NoneType' object is not callable, as my model does not have a tokenizer. Why is this?

Traceback (most recent call last): File "/home/InternLM-XComposer/inference_lora_demo.py", line 32, in response, his = model.chat(tokenizer, query, image, do_sample=False, num_beams=3, use_meta=True) File "/home/miniconda3/envs/InternLM/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/nvme1/huggingface/modules/transformers_modules/internlm/internlm-xcomposer2d5-7b/4aa81f2bbf20a9ddd4137dfe847c142adf07b652/modeling_internlm_xcomposer2.py", line 631, in chat inputs, immask, = self.interleav_wrap_chat(query, image, history=history, meta_instruction=meta_instruction, hd_num=hd_num) File "/nvme1/huggingface/modules/transformers_modules/internlm/internlm-xcomposer2d5-7b/4aa81f2bbf20a9ddd4137dfe847c142adf07b652/modeling_internlm_xcomposer2.py", line 259, in interleav_wrap_chat part_tokens = self.tokenizer( TypeError: 'NoneType' object is not callable

import torch from transformers import AutoModel, AutoTokenizer from peft import AutoPeftModelForCausalLM

torch.set_grad_enabled(False)

adapter_path = '/home/InternLM-XComposer/finetune/output/finetune_lora_2/checkpoint-120'

model = AutoPeftModelForCausalLM.from_pretrained( adapter_path, device_map="auto", torch_dtype=torch.float16, trust_remote_code=True ).cuda().eval()

tokenizer = AutoTokenizer.from_pretrained(adapter_path, trust_remote_code=True) model.tokenizer = tokenizer

query = 'Image1 ; Image2 ; Image3 ; I want to buy a car from the three given cars, analyze their advantages and weaknesses one by one' image = ['./examples/cars1.jpg', './examples/cars2.jpg', './examples/cars3.jpg',]

with torch.autocast(device_type='cuda', dtype=torch.float16): response, his = model.chat(tokenizer, query, image, do_sample=False, num_beams=3, use_meta=True) print(response)

yuhangzang commented 2 months ago

You may read the document and use the merge_peft_adapter.py to merge the LoRA weight.