Feature Request: Token Steraming via the API

KoboldAI / KoboldAI-Client

For GGUF support, see KoboldCPP: https://github.com/LostRuins/koboldcpp

https://koboldai.com

GNU Affero General Public License v3.0

3.51k stars 758 forks source link

Open InconsolableCellist opened 1 year ago

InconsolableCellist commented 1 year ago

Would it be possible to add token streaming to the API? Currently it seems this feature isn't exposed.

masonj19 commented 1 year ago

I would also like token streaming to be added to the API. I'm honestly surprised it hasn't already been added yet.