Open BBC-Esq opened 4 months ago
Seems to work fine here with the converted Llama-3-8B-Instruct:
You: Hello, how are you doing today?
Llama3:
I'm just a language model, I don't have emotions or feelings like humans do, so I don't have good or bad days. I'm always "on" and ready to assist you with any questions or tasks you may have! It's great to chat with you, though. How can I help you today?<|eot_id|>
You:
OK, let me retry it...thanks.
Strange...it did the same thing again.
Below I am including (1) the full response, (2) the command I used to run the script, (3) a modified script I created that (a) uses a hard-coded system message and path to the model instead, (b), uses the "apply_chat_template" method from transformers, (c) adds "exit" to exit the conversation, and (4) the output from my script, which is basically producing the same thing as your results...any idea?
Also, I'm getting these warning messages immediately before it states "You:" where I enter my prompt:
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Loading the model...
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
I solved the issue by using the "end_token" parameter. Here's the script for peoples' benefit:
class Llama38BInstructModel:
def __init__(self, user_prompt="PLACEHOLDER_FOR_USER_PROMPT", system_prompt="You are a helpful assistant who answers questions in a succinct fashion based on the contexts given to you. Only base your answer to the following question on the provided context/contexts accompanying this question. If you cannot answer based on the included context/contexts alone, please state so."):
self.user_prompt = user_prompt
self.system_prompt = system_prompt
self.model_dir = "PATH TO CONVERTED MODEL DIRECTORY"
self.model_name = os.path.basename(self.model_dir)
self.intra_threads = max(os.cpu_count() - 4, 4)
self.FLASH_ATTN = torch.cuda.is_available() and torch.cuda.get_device_capability()[0] >= 8.6
self.generator = ctranslate2.Generator(
self.model_dir,
device="cuda",
compute_type="int8",
flash_attention=self.FLASH_ATTN,
intra_threads=self.intra_threads
)
self.tokenizer = AutoTokenizer.from_pretrained("PATH TO CONVERTED MODEL DIRECTORY")
def build_prompt(self):
return f"system\n{self.system_prompt}user\n{self.user_prompt}assistant\n"
def generate_response(self):
prompt = self.build_prompt()
tokens = self.tokenizer.convert_ids_to_tokens(self.tokenizer.encode(prompt))
results_batch = self.generator.generate_batch(
[tokens],
include_prompt_in_result=False,
end_token="<|eot_id|>",
return_end_token=False,
max_batch_size=4095,
batch_type="tokens",
beam_size=1,
num_hypotheses=1,
max_length=512,
sampling_temperature=0.0,
)
output = self.tokenizer.decode(results_batch[0].sequences_ids[0])
print("\nGenerated response:")
print(output)
del self.generator
del self.tokenizer
torch.cuda.empty_cache()
gc.collect()
I ran the new Llama3 sample script and it seems to be conversing with itself so I think there's a problem with how the prompt is being constructed...See below: