Closed zhulinJulia24 closed 2 weeks ago
topk, topp操作会减少在做sample 之前 vocab的候选词的数量。当候选词减少到4个的时候,我觉得没必要非得输出5个词的概率(vllm那边会输出,不过概率是-inf)。如果输出的token不在前5的话,那么输出的top logrobs可能会有6个。
topk, topp操作会减少在做sample 之前 vocab的候选词的数量。当候选词减少到4个的时候,我觉得没必要非得输出5个词的概率(vllm那边会输出,不过概率是-inf)。如果输出的token不在前5的话,那么输出的top logrobs可能会有6个。
是的,但在这种情况下我试过top > 5还是能输出>=5的数字,所以我理解topk, topp操作后候选词应该是多于5个,那么top=5的话应该输出>=5个?
最少输出一个,最多输出top+1个(所选token不在top里面)
@zhulinJulia24 3.And I cannot get logprobs return by using /v1/chat/completions api. 请问第三条这个问题有解决办法吗
@zhulinJulia24 3.And I cannot get logprobs return by using /v1/chat/completions api. 请问第三条这个问题有解决办法吗
I think it supports on latest version. You can have a try
Checklist
Describe the bug
After set logprobs = true and top_logprobs = 5. The number of top logrobs response is not correct, only 4 for each token. I suppose it should be 5 for each token.
Reproduction
check the response, only 4 top probs return. I want 5 top probs![image](https://github.com/InternLM/lmdeploy/assets/145004780/7dddf7f4-6ba1-413b-9d2b-d8eea8f3570a)
logprobs in response is none, it's unexpected.
Environment
Error traceback
No response