Auto apply correct chat template when using OpenAI compatible API

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with KoboldAI's UI

https://github.com/lostruins/koboldcpp

GNU Affero General Public License v3.0

4.34k stars 310 forks source link

Auto apply correct chat template when using OpenAI compatible API #925

Closed Tureti closed 2 weeks ago

Tureti commented 2 weeks ago

v1 api currently uses a harcoded alpaca prompt template with support for specifying adapters to change it: https://github.com/LostRuins/koboldcpp/pull/466 This method is not supported in for example Open-WebUI. It would be great if koboldcpp could automatically apply the chat template specified in tokenizer.chat_template of the gguf model.

Tureti commented 2 weeks ago

Allowing to set a custom chat template for the v1 api when starting koboldcpp would be enough for a quick fix as well if that seems easier.

LostRuins commented 2 weeks ago

This is supported, see https://github.com/LostRuins/koboldcpp/wiki#what-is---chatcompletionsadapter

Tureti commented 2 weeks ago

Ok sorry I missed that.