Ability to draw multiple samples

[thomasahle]

Is your feature request related to a problem? Please describe. I'm trying to do something like Getting 50% (SoTA) on ARC-AGI with GPT-4o, and it requires drawing thousands of independent samples.

Describe the solution you'd like I'm basically trying to reopen https://github.com/jxnl/instructor/issues/576 since it seems it was closed together with the pull request https://github.com/jxnl/instructor/pull/578 because it wasn't clear "if people really needed this".

This means instructor should support the n=... argument as in:

client.chat.completions.create_with_completion(
        model="gpt-3.5-turbo-0613",
        response_model=Foo,
        messages=[{"role": "user", "content": f"Make a python program to solve the task: {data}"},
        ],
        n=3,
    )

which would return a list[Foo] with 3 objects.

Describe alternatives you've considered Say my output type is:

class Foo(BaseModel):
    solution: str

I could instead use

class Foo(BaseModel):
  solutions: Annotated[list[str], Len(min_length=5, max_length=5)]

to force the model to give me 5 outputs.

However, this might overflow the output context; and it would make the outputs "dependent" when what I want are independent samples.

I could also do n independent calls to client.chat.completions.create, but when I have a very large prompt, this is very expensive.

[jxnl]

I closed due to some breaking changes, but if it's interesting, would you be able and interested in reviving the PR.

[thomasahle]

I'm not familiar with the instructor codebase. But I'll have a look!

[thomasahle]

Ok, I think you are right that Anthropic has to be excluded, as they don't support it. I guess an option would be to just do multiple calls in this case, but maybe it's better to throw an error.

Otherwise, I don't see any issues with your PR!