search

anon998 / simple-proxy-for-tavern

GNU Affero General Public License v3.0
110 stars 6 forks source link

Abort generation on KoboldCPP when stop streaming button is pressed. #4

Closed ClassicDirt closed 1 year ago

ClassicDirt commented 1 year ago

During generation, when stop streaming is clicked, the stream is aborted but koboldcpp continues to generate text in the background. I think it would be preferable if koboldcpp generation is fully aborted when stop streaming is clicked.

anon998 commented 1 year ago

I haven't found a way to see that button being pressed from the proxy yet, but I made a few changes, and now the previous request is aborted when a new one comes. So you can press the stop button and generate again, and it will work. In theory, at least.

anon998 commented 1 year ago

I think I made it work with the latest commit now. https://github.com/anon998/simple-proxy-for-tavern/commit/eb8ccf91b3125494b30888469f2f4d8d8478c0f2