search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.66k stars 334 forks source link

SSE streaming end of stream character. #697

Closed Stargate256 closed 4 months ago

Stargate256 commented 6 months ago

Can I somehow have it transmit an end of stream character after the stream has ended?

SO basically I would like to have a special character sequence or just a single ASCII character that would be transmitted when token generation ends or it reaches a stop character.

If the feature doesn't exist could you please point me to the right direction of knowing which file must I modify and then recompile the project to get such a feature?

LostRuins commented 6 months ago

If you are using the OpenAI api, this is already done, it transmits a [DONE] token as seen here: https://github.com/LostRuins/koboldcpp/blob/concedo/koboldcpp.py#L695

For the KoboldAPI, it simply ends the stream when completed. You can modify the above line to add whatever you like.