search

ngruver / llmtime

https://arxiv.org/abs/2310.07820
MIT License
628 stars 139 forks source link

NLL function is missing for GPT #29

Open ditteaaroee opened 4 months ago

ditteaaroee commented 4 months ago

The gpt_nll_fn should be added for gpt-3.5 in nll_fns

mfinzi commented 4 months ago

Sadly the ability to get the NLLs only exists for gpt3 and not 3.5 or 4 within the OpenAI API. It can be computed approximately via sampling, but it will be prohibitively slow and costly. If you need NLLs you might have to switch to gpt-3 or one of the open source models.

ditteaaroee commented 4 months ago

What's the reason for that? I've been able to add hypertuning based on NLL's when using gpt3.5. However, there's a problem with the output length that seems to be varying and thus not aligning with target length.

mfinzi commented 4 months ago

OpenAI didn't enable it as part of the chat completion when we wrote the paper, but it looks like they actually have added log probs as a feature now as you point out, so it should be possible. (I wasn't aware they had made the change). Time is stretched a bit thin for me at the moment, but we'd definitely be happy to review a pull request with this new feature. What's the challenge with the output length and target length exactly?