search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.66k stars 334 forks source link

How do i enable streaming in chat mode (aesthetic chat ui) #29

Closed x-legion closed 1 year ago

LostRuins commented 1 year ago

put streaming in the url, e.g. http://localhost:5001?streaming=1 and it should work.

henk717 commented 1 year ago

In addition it only shows streaming text when you are not using the Aestethic Text UI.

x-legion commented 1 year ago

In addition it only shows streaming text when you are not using the Aestethic Text UI.

exactly.

x-legion commented 1 year ago

put streaming in the url, e.g. http://localhost:5001?streaming=1 and it should work.

but it doesnt work on aesthetic chat ui

LostRuins commented 1 year ago

Yes, currently streaming does not work in the aesthetic chat UI... it may be added in future but is not a high priority for now.

Enferlain commented 1 year ago

@LostRuins By the way, I haven't checked since last time, but do you know if streaming is possible through sillytavern? I think oobabooga works with streaming, not sure about koboldcpp though.

LostRuins commented 1 year ago

It depends on how they implemented streaming. The Kobold API emulates it by breaking it up into small requests - this is not true streaming which is not implemented.

LostRuins commented 1 year ago

True streaming is now available in newest versions.