Advanced Evals: Combining Multiple Annotators of Varying Quality

nelsonauner commented 1 month ago

Summary

This summary adds a notebook for dealing with multiple annotators, partially replicating the seminal MT-Bench paper while using OpenAI's new structured outputs and token probability features unavailable in other LLMs

Motivation

There currently is no cookbook on dealing with combining a large number of human annotators with LLM-As-Judge, even though this practice is quite common.

For new content

When contributing new content, read through our contribution guidelines, and mark the following action items as completed:

[x] I have added a new entry in registry.yaml (and, optionally, in authors.yaml) so that my content renders on the cookbook website.
[x] I have conducted a self-review of my content based on the contribution guidelines:
- [x] Relevance: This content is related to building with OpenAI technologies and is useful to others.
- [x] Uniqueness: I have searched for related examples in the OpenAI Cookbook, and verified that my content offers new insights or unique information compared to existing documentation.
- [x] Spelling and Grammar: I have checked for spelling or grammatical mistakes.
- [x] Clarity: I have done a final read-through and verified that my submission is well-organized and easy to understand.
- [x] Correctness: The information I include is correct and all of my code executes successfully.
- [x] Completeness: I have explained everything fully, including all necessary references and citations.

We will rate each of these areas on a scale from 1 to 4, and will only accept contributions that score 3 or higher on all areas. Refer to our contribution guidelines for more details.

nelsonauner commented 3 weeks ago

@pap-openai Anything I can do to help get this reviewed (or close it out?)

pap-openai commented 2 weeks ago

@pap-openai Anything I can do to help get this reviewed (or close it out?)

Hi @nelsonauner, we're currently working on a contributor license agreement and will come back a bit later once we've figured that out

openai / openai-cookbook