Decode problem of 13B

[WuNein]

Run in nvidia pytorch docker 23.05, with rtx a6000 48G

from transformers import LlamaForCausalLM, LlamaTokenizer  
import torch

ckpt = '/root/hdd/WizardLM-13B-V1.0'
device = torch.device('cuda')
# model = LlamaForCausalLM.from_pretrained(ckpt, device_map='auto',  load_in_8bit = True)
model = LlamaForCausalLM.from_pretrained(ckpt, device_map='auto',  torch_dtype=torch.float16)
tokenizer = LlamaTokenizer.from_pretrained(ckpt)
tokenizer.pad_token_id = (
        0  # unk. we want this to be different from the eos token
    )
tokenizer.padding_side = "left"
model.half()  # seems to fix bugs for some users.
model.eval()

if torch.__version__ >= "2":
    model = torch.compile(model)
prompt = f"A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {review} ASSISTANT: "
from transformers import GenerationConfig

model.config.pad_token_id = tokenizer.pad_token_id = 0  # unk
model.config.bos_token_id = 1
model.config.eos_token_id = 2

inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs["input_ids"].to('cuda')
max_new_tokens=50
generation_config = GenerationConfig(
    temperature=1,
            top_p=0.95,
            top_k=40,
            num_beams=1,
)
with torch.no_grad():
    generation_output = model.generate(
        input_ids=input_ids,
        generation_config=generation_config,
        return_dict_in_generate=True,
        output_scores=True,
        max_new_tokens=max_new_tokens,
    )
s = generation_output.sequences       
output = tokenizer.batch_decode(s, skip_special_tokens=True)
output = output[0].split("ASSISTANT:")[1].strip()

The output is a mass! accom [[⁄ recherReceti kinds TODO��志Stats charact zus Probably weiteren��志Stats charact zus Probably weiteren��志Stats charact zus Probably weiteren��志Stats charact zus Probably weiteren��志Stats charact zus Probably weiteren��志Stats charact zus Probably weiteren 可能的问题是什么？ I double check the checkpoint, it seems fine.

[WuNein]

Do i need python src/weight_diff_wizard.py recover --path_raw --path_diff --path_tuned ?

[cornleaf]

Same as u ,however no feedback from the owner!

[WuNein]

Same as u ,however no feedback from the owner!

I fixed, i forget to merge the weight.