openai / evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Other
14.76k stars 2.58k forks source link

Getting started example doesn't work - oieval attempts to update a None type object #1515

Closed jswang closed 6 months ago

jswang commented 6 months ago

Describe the bug

When running the example provided in https://github.com/openai/evals/blob/main/docs/run-evals.md: oaieval gpt-3.5-turbo test-match, the following error is encountered out of the box:

 File "/Users/juliewang/Documents/evals/evals/cli/oaieval.py", line 167, in run
    eval_spec.args.update(extra_eval_params)
    ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'update'

This is due to this line: https://github.com/openai/evals/blob/main/evals/cli/oaieval.py#L158. The args field of EvalSpec is optional: https://github.com/openai/evals/blob/0dc0ba455194f5c9c4566ce415a2b2714f56e2db/evals/base.py#L58, and in the test-match case, eval_spec.args is None.

To Reproduce

  1. Run oaieval gpt-3.5-turbo test-match

Code snippets

No response

OS

macOS

Python version

Python 3.12.2

Library version

openai-evals commit b5853eb4fed5001188671e87b20d8f8261b57e63

thesofakillers commented 6 months ago

Thx for raising this! I've opened a PR that should address it.