Closed DePasqualeOrg closed 2 months ago
Per the tokenizer config, the end of text is:
"eos_token": "<|endoftext|>",
It looks like people are manually modifying the config or manipulating the runtime. Ideally the shipping tokenizer config would indicate the correct eos tokens.
That said, the swift-transformers code only allows a single eos token.
We have a couple options here:
<|end|>
in the tokenizer config -- I don't know if this is actually correctIn Python, we do a combination of 1 (change the config) and 3. The argument is nice because it adds a lot of flexibility. Regarding 2, it's not clear to me if the original model authors intended the model to have two EOS tokens or if it was just an oversight not to update it in the tokenizer...
OK, then let's plan on adding #3 -- that will make it flexible at least.
OK, I am working on the model configuration for #53 so it will be included with that.
This Phi 3 model used in the LLMEval app doesn't behave as expected. It looks like the stop token is not being recognized.
Prompt:
Output:
Related issues: https://github.com/ggerganov/llama.cpp/issues/6903 https://github.com/nomic-ai/gpt4all/issues/2271 https://github.com/huggingface/swift-transformers/issues/98