Using `make-llm-openai-compatible` with Mistral AI fails parsing the partial responses

[KaiHa]

I have configured Mistral AI (https://docs.mistral.ai/api/) as openai-compatible provider and was observing empty partial responses in the log. I debugged it until this[1] point and wondered why a regex is used for handling the JSON response. The response from Mistral AI looks like following:

data: {\"id\":\"00000000000000000\",\"object\":\"chat.completion.chunk\",\"created\":1710964073,\"model\":\"open-mixtral-8x7b\",\"choices\":[{\"index\":0,\"delta\":{\"role\":null,\"content\":\"I don\"},\"finish_reason\":null}],\"usage\":null}

I was able to get the response by fiddling with the regex[1] but I was wondering if it would be more robust if something like json-read-from-string would be used.

I am using llm version 0.12.0 with Emacs 29.2.

And thanks for providing this library, so that I don't have to leave Emacs to participate in this LLM craze :)

[1] https://github.com/ahyatt/llm/blob/5fb10b9bdbfc6bb384c14fbfc8caf09fdb4b99e8/llm-openai.el#L271C38-L271C51

[ahyatt]

Thanks for the report. I'm using a regex because I didn't want to parse all the strings, since most of them we've already read. Plus, there isn't always JSON on every line. Perhaps that optimization isn't worth it - certainly if the regex isn't robust enough, it deserves a fix.

There is a fix coming soon with using plz as a backend, with actually support for the event-stream format that Open AI uses. That one's the best solution, and will fix this. In the meantime, it's worth seeing what I can do for a quicker fix, so let me work on that.