brian-pieces
commented
3 months ago Like Rosie said above, seems to be a common issue with Phi-3 and larger context windows:
- https://github.com/microsoft/Phi-3CookBook/issues/45
- https://www.reddit.com/r/LocalLLaMA/comments/1ccbg9x/phi3_rag/
- https://www.reddit.com/r/LocalLLaMA/comments/1cbt78y/comment/l10xvms/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
Given that our Live Context usually has anywhere from 2k-5k context, you're likely to experience gibberish output with both Phi-3 Mini 4k and 128k.
We're actively researching solutions. Thank you for the patience!
Software
Desktop Application
Operating System / Platform
macOS
Your Pieces OS Version
10.0.0
Early Access Program
Kindly describe the bug and include as much detail as possible on what you were doing so we can reproduce the bug.
Several users reporting that when using Phi-3 especially with Live Context, it produces gibberish. We've done some research on this and it looks like this is a common issue when using Phi-3 with RAG. We will continue investigating but in the meantime Phi-3 should work just fine for interactions where you haven't added context but if you wanted to use context you should try a different local model like Mistral or Granite!