`top_p` messes up `top_logprobs`

[thomas-schillaci]

System Info

tgi docker image 2.0.4

Information

• [X] Docker
• [ ] The CLI directly

Tasks

• [X] An officially supported command
• [ ] My own modifications

Reproduction

Passing this body to the chat completions route:

{ "logprobs": true, "max_tokens": 1, "messages": [ { "role": "user", "content": "What is Deep Learning?" } ], "model": "mistralai/Mistral-7B-Instruct-v0.2", "top_logprobs": 3, "top_p": 0.5 }

Gives:

{ "id": "", "object": "text_completion", "created": 1719230167, "model": "/data/models--meta-llama--Meta-Llama-3-8B-Instruct/snapshots/e1945c40cd546c78e41f1151f4db032b271faeaa", "system_fingerprint": "2.0.4-sha-f426a33", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Deep" }, "logprobs": { "content": [ { "token": "Deep", "logprob": 0, "top_logprobs": [ { "token": "Deep", "logprob": 0 } ] } ] }, "finish_reason": "length" } ], "usage": { "prompt_tokens": 15, "completion_tokens": 1, "total_tokens": 16 } }

Whereas when removing top_p, the output is (look at top_logprobs):

{ "id": "", "object": "text_completion", "created": 1719230253, "model": "/data/models--meta-llama--Meta-Llama-3-8B-Instruct/snapshots/e1945c40cd546c78e41f1151f4db032b271faeaa", "system_fingerprint": "2.0.4-sha-f426a33", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Deep" }, "logprobs": { "content": [ { "token": "Deep", "logprob": -0.005470276, "top_logprobs": [ { "token": "Deep", "logprob": -0.005470276 }, { "token": "A", "logprob": -5.2382812 }, { "token": "**", "logprob": -9.1015625 } ] } ] }, "finish_reason": "length" } ], "usage": { "prompt_tokens": 15, "completion_tokens": 1, "total_tokens": 16 } }

Expected behavior

Setting top_p should not alter the value of top_logprobs in the response.

[LysandreJik]

Hey @thomas-schillaci, thanks for the report! Could you put the contents of your request to the so I can see why it's acting up?

[thomas-schillaci]

@LysandreJik it is already in the issue, or do you need anything else?

[alexsniffin]

Hello, I'm also running into this problem. If you set logprobs as true, top_logprobs greater than 1 and any value for top_p, it will only return only one logprob. When I'd expect it to equal top_logprobs which should be greater than 1.