search

intel-analytics / ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, GraphRAG, DeepSpeed, Axolotl, etc
Apache License 2.0
6.7k stars 1.26k forks source link

Falcon-7B model generate unusual result #8915

Closed jenniew closed 1 year ago

jenniew commented 1 year ago

Use bigdl llm to load Falco-7b in bf16, and generate output string with size of 256 using input of 256 tokens. The output string ends with several strange characters. Code:

model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
input_ids = tokenizer.encode(input_str, return_tensors="pt")
while input_ids.shape[1] > input_size:
      input_str = input_str[:-1]
      input_ids = tokenizer.encode(input_str, return_tensors="pt")
output = model.generate(input_ids, do_sample=False, max_new_tokens=output_size, use_cache=True)
output_str = tokenizer.decode(output[0], skip_special_tokens=True)

Output: The new CEO sat on the edge of his desk and stared into the middle distance. The other executives had already left. Only he, his secretary and two security guards remained. Above them hung a black velvet curtain embroidered with silver threads. A shining diamond of light from a single window in the far wall was reflected by this curtain, sending out dazzling rays as it swept across the floor. The CEO stared at this scene for some time before he spoke:\n“This is my new office.”\nThe secretary looked up from his typing and glanced around the room to confirm that he did indeed mean it. She then smiled in anticipation of the things she would do here when her boss was not present. The two security guards stood together at attention by the door, awaiting instructions. They had been hired for their appearance as much as for their skills.\n“I am very proud to have you all working with me,” said the CEO, “and I look forward to our time spent here.” He paused for a moment before continuing:\n“Before we start on our mission of world domination, though, we need to decide what we are going to do about the current crisis.”\n“What crisis?” asked the secretary. The CEO looked at him and said:\n“The crisis of the world economy.”\n“““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““““?

jenniew commented 1 year ago

After updated with latest benchmark test code, the issue disappeared. The issue may be caused by incorrect bf16 test code.