[x] I reviewed the Discussions, and have a new bug or useful enhancement to share.
Expected Behavior
All words in the prompt and completions should be used.
Current Behavior
I ported the example code from batched.swift in the repo. I noticed that certain words like "interested", "Francisco", and other random words are getting skipped in the tokenizer both in the prompt and in the response. This makes the completions seem nonsensical. The original example project in the repo has the same issue.
Environment and Context
M2 Max MacBook Pro (32GB), M2 Mac Mini (24GB)
macOS 14.0
Swift (macOS app) via Swift Package
Failure Information (for bugs)
Please help provide information about the failure / bug.
I used the same token code from the example project.
Steps to Reproduce
Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.
Run the example batched.swift project.
I used LLaMA-2-13B-Chat
Use the prompt: "Transcript of a text message, where Ethan interacts with his virtual assistant, Nova. Nova is helpful, kind, honest, good at writing, and never fails to answer Ethan's requests immediately and with precision.\nETHAN: Hello, what can you tell me about Cupertino and it's surounding areas? I'm interested in museums and zoos.\n"
Prerequisites
Please answer the following questions for yourself before submitting an issue.
Expected Behavior
All words in the prompt and completions should be used.
Current Behavior
I ported the example code from batched.swift in the repo. I noticed that certain words like "interested", "Francisco", and other random words are getting skipped in the tokenizer both in the prompt and in the response. This makes the completions seem nonsensical. The original example project in the repo has the same issue.
Environment and Context
M2 Max MacBook Pro (32GB), M2 Mac Mini (24GB) macOS 14.0 Swift (macOS app) via Swift Package
Failure Information (for bugs)
Please help provide information about the failure / bug.
I used the same token code from the example project.
Steps to Reproduce
Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.
"Transcript of a text message, where Ethan interacts with his virtual assistant, Nova. Nova is helpful, kind, honest, good at writing, and never fails to answer Ethan's requests immediately and with precision.\nETHAN: Hello, what can you tell me about Cupertino and it's surounding areas? I'm interested in museums and zoos.\n"
Logs