Compute perplexity/logits for the prompt

[dsmilkov]

I'd like to use Phi-2 to compute perplexity of the prompts over an entire dataset. Is there an API for this? In the short term, I'm happy to fork https://github.com/vllm-project/vllm/blob/d0215a58e78572d91dadafe9d832a2db89b09a13/vllm/model_executor/models/phi_1_5.py if you provide pointer on how to do that.

Also happy to later contribute back an API that works for all causal models.

[mayiran1999]

I have the same need. Have anyone found a possible way to get the logits of the prompt?

[caiyuhu]

I have the same need too, but unfortunately it appears that vLLM has not yet implemented support for it, as evidenced by the following issue discussion. https://github.com/vllm-project/vllm/issues/185

[lekhang4497]

I think you can use the parameter prompt_logprobs in SamplingParams for this purpose.

1328

[dylanbowman314]

prompt_logprobs can only return the probabilities for the top <=20 tokens right now, so not applicable for this usecase.

[junzhang-zj]

Is there any progress on this issue at the moment?

[Tendo33]

same issue here

[CodeAsPoetry]

you can set logprobs=1, prompt_logprobs=1. Then,

[CodeAsPoetry]

test prompt >20 , maybe ok