search

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://UKGovernmentBEIS.github.io/inspect_ai/
MIT License
385 stars 41 forks source link

Error when getting logprobs for EOF token from OpenAI API #42

Closed lukasberglund closed 3 weeks ago

lukasberglund commented 3 weeks ago

Sometimes, when I try to get top logprobs via the OpenAI API. I get the error pictured below. The issue seems to be that the OpenAI sometime returns a logprobs object for which the bytes field is None, e.g. {'token': '<|end|>', 'bytes': None, 'logprob': -28.681221},, but pydantic expects the bytes field to be a list. I think this could be fixed by changing the type signature for LogProbs in inspect, but I'm not sure. (I tried doing this and for some reason pydantic was still complaining, maybe I made some mistake though.)

Here's the full log file

image image image
aisi-inspect commented 3 weeks ago

Yes, this should be a simple schema fix! Do you think it would be legit for us to covert the None into an empty list? (just so that readers of the log can depend on having a list vs. having to check for None).

aisi-inspect commented 3 weeks ago

This should be resolved with this commit: https://github.com/UKGovernmentBEIS/inspect_ai/commit/8141ef25f2eef2cb23c8ac8a4d2ee02872db6549 (I added None to the schema for bytes)