search

agnaistic / agnai

AI Agnostic (Multi-user and Multi-bot) Chat with Fictional Characters. Designed with scale in mind.
https://agnai.chat
GNU Affero General Public License v3.0
535 stars 107 forks source link

[Feature Request] Allow token streaming from Kobold(cpp) #379

Closed SammCheese closed 1 year ago

SammCheese commented 1 year ago

Some models might take a while to generate on some hardware. So to not always have 3 dots hovering around for half an hour, it might be beneficial to implement token streaming similar to Kobold lite's implementation. It would severely reduce idle waiting times for the user and benefit the user experience.

sceuick commented 1 year ago

Need to wait for Kobold to implement an API that supports it. I'll likely need to be notified when that happens. Happy to implement when an server-side events API is available in Kobold.

Thangzy commented 1 year ago

Hey !

Just wanted to let you know that @SammCheese added a compatible endpoint for agnaistic in their latest version https://github.com/LostRuins/koboldcpp/releases/tag/v1.30.3

Thank you for your work !

EDIT: credits is important kids

SammCheese commented 1 year ago

That was me xd

Thangzy commented 1 year ago

Oh my bad ahah, thank you for doing it !

SammCheese commented 1 year ago

Implemented in 0fd4b58