feat: Nonstreaming API

[JNeuvonen]

The implementation uses the same start function inside process.rs for multithreading but just doesn't send server events back to the request sender on every new token but collects the tokens into a string buffer.

Currently, there is no client-side implementation, so merging should not affect client-side at all. Next, we could open an issue for client-side implementation as well.

Here is a request body for quickly testing the API (stream flag is false):

{"sampler":"top-p-top-k","prompt":"AI: Greeting! I am a friendly AI assistant. Feel free to ask me anything.\nHuman: Hello world\nAI: ","max_tokens":200,"temperature":1,"seed":147,"frequency_penalty":0.6,"presence_penalty":0,"top_k":42,"top_p":1,"stop":["AI: ","Human: "],"stream":false}

Issue

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
local-ai-web	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 24, 2023 4:14am

[louisgv]

@JNeuvonen hey sorry about the slow review on my end, I've been pretty busy with summer chores/errand and also other works xD... Also was investigating #62 and why the upstream llama metal doesn't seem to work on Mac anymore :d..... Will get to this by Wednesday.

Is it ok for me to cook it up a bit if I find something wrong/missing, or would you prefer just comment and you can take care of it? LMK what type of feedback is cool for you :)

[JNeuvonen]

Thanks, no problem at all, totally understandable. Comment & let me figure it out would be preferred feedback form, but if it’s very simple changes you can do it as well.

On Mon 31. Jul 2023 at 23.36, L @.***> wrote:

@JNeuvonen https://github.com/JNeuvonen hey sorry about the slow review on my end, I've been pretty busy with summer chores/errand and also other works xD... Also was investigating #62 https://github.com/louisgv/local.ai/pull/62 and why the upstream llama metal doesn't seem to work on Mac anymore :d..... Will get to this by Wednesday.

Is it ok for me to cook it up a bit if I find something wrong/missing, or would you prefer just comment and you can take care of it? LMK what type of feedback is cool for you :)

— Reply to this email directly, view it on GitHub https://github.com/louisgv/local.ai/pull/85#issuecomment-1659126502, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARW4OHJ6BECN5BFOGVFRKYTXTAJNZANCNFSM6AAAAAA2VF4LWU . You are receiving this because you were mentioned.Message ID: @.***>

[louisgv]

@JNeuvonen invited you as repo collaborator

[louisgv]

@JNeuvonen lmk if you're still able to update the PR - otherwise I can get on it sometime next week!

[JNeuvonen]

Hey, I apologize that I didn't come back earlier. Back when I was working on this, I was on a summer vacation, now I am back on my work schedule, and I have less time & focus. Please feel free to finish the feature.