search

exo-explore / exo

Run your own AI cluster at home with everyday devices 📱💻 🖥️⌚
GNU General Public License v3.0
6.56k stars 342 forks source link

BUG: Spaces between words are not streamed #164

Closed vlbosch closed 3 weeks ago

vlbosch commented 3 weeks ago

When using the TinyChat interface or connecting to exo via the HTTP API with for example TypingMind, the spaces between words are lost. I do see the spaces in the terminal environment, so I suppose the model is generating a correct response. It just seems to get lost when streaming to the client.

AlexCheema commented 3 weeks ago

When using the TinyChat interface or connecting to exo via the HTTP API with for example TypingMind, the spaces between words are lost. I do see the spaces in the terminal environment, so I suppose the model is generating a correct response. It just seems to get lost when streaming to the client.

Hey, what devices are you running on? What commit of exo are you using?

vlbosch commented 3 weeks ago

Hi! I am running exo on a M3 Max MacBook Pro with 128GB RAM with macOS 15.0 DP5 and the latest commit – [6db73fa]. Let me know if you need any other info

AlexCheema commented 3 weeks ago

Which model? I have the same setup and seems fine.

Screenshot 2024-08-22 at 12 53 31
vlbosch commented 3 weeks ago
Scherm­afbeelding 2024-08-22 om 13 56 18

I am running Mistral Large. Here's a screenshot of the terminal and the web-ui.

AlexCheema commented 3 weeks ago

Interesting.. Are you on Discord? I'd like to investigate some more and fix this for you.

AlexCheema commented 3 weeks ago

The next thing to check would be the network tab in your browser to see what is being sent over the wire.

AlexCheema commented 3 weeks ago

It's probably a tokenizer issue.

vlbosch commented 3 weeks ago

Yes I am :-)

AlexCheema commented 3 weeks ago

Let me know if this is fixed @vlbosch

vlbosch commented 3 weeks ago

I can confirm it works both in the Tiny webui as well as with the OpenAI compatibel HTTP API in Typing Mind. Thank you for this very fast fix! 😃🙏🏻

AlexCheema commented 3 weeks ago

I can confirm it works both in the Tiny webui as well as with the OpenAI compatibel HTTP API in Typing Mind. Thank you for this very fast fix! 😃🙏🏻

Awesome!

Please let me know if you run into any other issues, have suggestions for any improvements, or general comments. In particular, if you're annoyed by something or frustrated by something, please make an issue - it's really valuable to get this feedback so we can improve!