braintrustdata / braintrust-sdk

Apache License 2.0
23 stars 13 forks source link

braintrust's `wrap_openai` and instructors `patch` aren't compatible #430

Open mattflo opened 2 hours ago

mattflo commented 2 hours ago

At one point this probably worked as outlined in the documentation: https://www.braintrust.dev/docs/guides/tracing#instructor

Monkey patching code is treacherous...

I'm on the latest version of the relevant packages:

braintrust         0.0.167
braintrust-core    0.0.54
openai             1.53.0
instructor         1.6.3

Here's how I create the client:

instructor_wrapped_client = instructor.patch(OpenAI())
client = wrap_openai(instructor_wrapped_client)

Here's how the client is used:

    return client.chat.completions.create(
        model="gpt-4o-mini",
        response_model=DiscountRequest,
        messages=[
            {
                "role": "system",
                "content": system_prompt,
            },
            {
                "role": "user",
                "content": user_prompt,
            },
        ],
    )

Here's the error:

   E           instructor.exceptions.InstructorRetryException: 'LegacyAPIResponse' object has no attribute 'choices'

../../Library/Caches/pypoetry/virtualenvs/discount-email-response-fgEnZF0Y-py3.11/lib/python3.11/site-packages/instructor/retry.py:164: InstructorRetryException
ankrgyl commented 2 hours ago

Thank you so much for filing. We'll take a look!

mattflo commented 1 hour ago

Here's some more visibility into the callstack when the error occurs:

discount_email/src/discount.py:38: in generate
    return client.chat.completions.create(
../../Library/Caches/pypoetry/virtualenvs/discount-email-response-fgEnZF0Y-py3.11/lib/python3.11/site-packages/braintrust/oai.py:333: in create
    return ChatCompletionWrapper(self.__completions.with_raw_response.create, None).create(*args, **kwargs)
../../Library/Caches/pypoetry/virtualenvs/discount-email-response-fgEnZF0Y-py3.11/lib/python3.11/site-packages/braintrust/oai.py:104: in create
    create_response = self.create_fn(*args, **kwargs)
../../Library/Caches/pypoetry/virtualenvs/discount-email-response-fgEnZF0Y-py3.11/lib/python3.11/site-packages/openai/_legacy_response.py:356: in wrapped
    return cast(LegacyAPIResponse[R], func(*args, **kwargs))
../../Library/Caches/pypoetry/virtualenvs/discount-email-response-fgEnZF0Y-py3.11/lib/python3.11/site-packages/instructor/patch.py:188: in new_create_sync
    response = retry_sync(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _