search

asahi417 / lmppl

Calculate perplexity on a text with pre-trained language models. Support MLM (eg. DeBERTa), recurrent LM (eg. GPT3), and encoder-decoder LM (eg. Flan-T5).
MIT License
134 stars 11 forks source link

ppl in openai model #3

Closed XuandongZhao closed 1 year ago

XuandongZhao commented 1 year ago

nll.append(sum([i for i in completion['choices'][0]['logprobs']['token_logprobs'] if i is not None])) I think this calculation may be wrong. May need to change to

nll.append(sum([i for i in completion['choices'][0]['logprobs']['token_logprobs'] if i is not None]) / len([i for i in completion['choices'][0]['logprobs']['token_logprobs'] if i is not None]))

asahi417 commented 1 year ago

Thanks! I think you're right. I'll fix it accordingly!