openai / evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Other
14.35k stars 2.54k forks source link

`OpenAIChatCompletionFn` should `__init__` should accept `**kwargs` #1493

Open ezraporter opened 3 months ago

ezraporter commented 3 months ago

Describe the bug

Currently the __init__ method for OpenAICompletionFn has **kwargs but OpenAIChatCompletionFn does not. This creates problems if I have both:

In these cases I would like to do:

oaieval my_completion_fn,gpt-4-turbo-preview my_eval --completion_args my_param=my_value

But I get:

TypeError: OpenAIChatCompletionFn.__init__() got an unexpected keyword argument 'my_param'

To Reproduce

Run oaieval gpt-3.5-turbo test-match --completion_args ignore_me=1

Code snippets

No response

OS

macOS

Python version

Python v3.11.7

Library version

openai-evals 2.0.0.post1