search

openvinotoolkit / openvino.genai

Run Generative AI models using native OpenVINO C++ API
Apache License 2.0
69 stars 106 forks source link

`chat_sample.exe` doesn't work well when using `Llama-2-7b-chat-hf` #557

Open yviansu opened 4 days ago

yviansu commented 4 days ago

Error detail

When running chat_sample.exe using Llama-2-7b-chat-hf, seems that it always have a built-in question: image

Steps to reproduce

  1. Get Llama-2-7b-chat-hf models by this command: 
    optimum-cli export openvino --model "meta-llama/Llama-2-7b-chat-hf" --trust-remote-code "meta-llama/Llama-2-7b-chat-hf"
  2. Build the code by https://github.com/openvinotoolkit/openvino.genai/blob/releases/2024/2/src/docs/BUILD.md#build-instructions-1
  3. Run the command: 
    chat_sample.exe C:\models\Llama-2-7b-chat-hf

By the way, I also tested chat_sample.exe with TinyLlama-1.1B-Chat-v1.0, and it works well: image