search

microsoft / Phi-3CookBook

This is a Phi-3 book for getting started with Phi-3. Phi-3, a family of open AI models developed by Microsoft. Phi-3 models are the most capable and cost-effective small language models (SLMs) available, outperforming models of the same size and next size up across a variety of language, reasoning, coding, and math benchmarks.
MIT License
1.23k stars 114 forks source link

Error about tokenizer? #82

Open xushilin1 opened 3 days ago

xushilin1 commented 3 days ago

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

tokenizer.decode(tokenizer('<|user|>\nhello').input_ids)

The output of this line of code is <|user|> hello. Why not <|user|>\nhello

Expected/desired behavior

Versions

transformers==4.41.2

Mention any other details that might be useful

Thanks! We'll be in touch soon.

leestott commented 3 days ago

hi @xushilin1 can you please confirm which sample or tutorial this error is in?