LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
5.24k stars 360 forks source link

logprobs are ignored in the v1 API #576

Open Huge opened 10 months ago

Huge commented 10 months ago

OpenAI (re)added support for outputting probabilities of generated tokens to most of their LLMs: https://cookbook.openai.com/examples/using_logprobs

logprobs are ignored in my calls to http://localhost:5001/api#/v1/post_v1_generate

{
  "max_context_length": 2048,
  "max_length": 20,
  "prompt": "Niko the kobold stalked carefully down the alley, his small scaly figure obscured by a dusky cloak that fluttered lightly in the cold winter breeze.",
  "quiet": false,
  "rep_pen": 1.1,
  "rep_pen_range": 256,
  "rep_pen_slope": 1,
  "temperature": 0.5,
  "tfs": 1,
  "top_a": 0,
  "top_k": 100,
  "top_p": 0.9,
  "typical": 1,
  "logprobs": 1,
  "top_logprobs": 3
}

->

{
  "results": [
    {
      "text": "\n\nAs he approached the end of the narrow passageway, he paused and listened intently for"
    }
  ]
}
LostRuins commented 10 months ago

Token probabilities are currently not available over the API. KoboldCpp only implements a core subset of all openai methods.

Huge commented 10 months ago

Alright. During my "investigation" I have noticed one bug, that the input schema disappears when "Try it out" button.

And https://github.com/swagger-api/swagger-ui/issues/6528#issuecomment-785837911 would be great to enable experimenting by default, when the bug above can be fixed.