Function does not obey Enums

[chengyjonathan]

Describe the bug I set a enum for one of the function inputs. I have a pydantic class that refers to the enum. The output args show that the enum is not followed.

Expected behavior I would expect that the generated args obey the enum I set for that field.

[jxnl]

can you produce an example and show me your prompts? try out https://jxnl.github.io/instructor/tips/

[alexclaydon]

Apologies for commenting on a closed issue, but I'm seeing this consistently on all of the examples I've run so far which make use of enums.

I've tried the following:

• the query planner example from the web docs;
• the more detailed query planner execution example; and
• a range of input queries, from quite complex down to very simple (and including the example input queries from the docs); and
• the instructions here (also linked above) to try to coerce the model.

In each case, the failure mode was the same: Pydantic validation errors on the enum field. It's pretty consistently trying to use strings like "root", "node", "intermediate", "leaf", "end", "query", "calculation", etc, apparently ignoring the contents of the enum entirely.

I'm on Python 3.11.6, instructor 0.2.8, pydantic_2.4.2 and pydantic_core 2.10.1. I tried executing both through a Jupyter notebook and as a script from the command line.

Appreciate any guidance you might be able to provide! Please let me know if there's any additional information I can provide which would be helpful.

[chengyjonathan]

I want to note that this newly started failing yesterday without changes in my stack.

So not quite sure what that means

[chengyjonathan]

Experiencing the same yesterday. Enums repeatedly ignored

[jxnl]

i wonder if models changed... @chengyjonathan @alexclaydon I'll rerun my examples as tests next time. also have you guys tried the new retry logic by adding the

max_retries=2

It'll cost more tokens., but it can patch while i see if i can improve my prompts

[alexclaydon]

Thanks Jason, yes I actually just came back with the intention to add that information to my original comment: these validation errors are happening for me even when using a high number of retries (I tested up to 5) and GPT-4. The consistently is striking - it's doesn't seem to be intermittent. I haven't managed to do a successful run when using enums even with low-complexity code and input questions. That says to me that it might not be an issue requiring additional coercion of the LLM, but rather something more deterministic happening with Pydantic.

[jxnl]

@alexclaydon do you have a specifc gist i can use as a quick eval? I wonder if its how deserialization to json is. or if literal string could help too

[alexclaydon]

Thanks Jason - I’m not at my desk atm but could send you one this afternoon APAC. For now, though, if you run the query planner execution example as a script from the command line it should fail with those errors (assuming same deps).

[alexclaydon]

Update:

Just tried the following:

• Freezing the model at "gpt-4-0613": No change to result
• Adding a catchall other field to the enum: No change to result.
• Took the example in the docs here and changed:

    node_type: QueryType = Field(
        default=QueryType.SINGLE_QUESTION,
        description="Choose either 'single_question' or 'merge_multiple_responses' depending on the type of node.",
    )

to:

    node_type: str = Field(
        ...,
        description="Choose either 'single_question' or 'merge_multiple_responses' depending on the type of node.",
    )

This allowed me to run the code without it throwing validation errors, but I got the following result:

{
    "query_graph": [
        {
            "id": 1,
            "question": "What is the population of Canada?",
            "dependancies": [],
            "node_type": "root"
        },
        {
            "id": 2,
            "question": "What is Jason's home country?",
            "dependancies": [],
            "node_type": "root"
        },
        {
            "id": 3,
            "question": "What is the population of Jason's home country?",
            "dependancies": [
                2
            ],
            "node_type": "intermediate"
        },
        {
            "id": 4,
            "question": "What is the difference in populations of Canada and the Jason's home country?",
            "dependancies": [
                1,
                3
            ],
            "node_type": "end"
        }
    ]
}

It still seems to be ignoring the description attribute on that field and trying to use "root", "intermediate", "end" - similar to the strings it was passing to the enum previously.

[chengyjonathan]

I just want to add that I'm noticing the same thing. The enum-description fields no longer get populated. Which is interesting.

On Wed, Nov 1, 2023 at 7:54 PM Alex Claydon @.***> wrote:

Update:

Just tried the following:

• Freezing the model at "gpt-4-0613": No change to result

• Took the example in the docs here https://jxnl.github.io/instructor/examples/planning-tasks/ and changed:

  node_type: QueryType = Field( default=QueryType.SINGLE_QUESTION, description="Choose either 'single_question' or 'merge_multiple_responses' depending on the type of node.", )

to:

node_type: str = Field(
    ...,
    description="Choose either 'single_question' or 'merge_multiple_responses' depending on the type of node.",
)

This allowed me to run the code without it throwing validation errors, but I got the following result:

{ "query_graph": [ { "id": 1, "question": "What is the population of Canada?", "dependancies": [], "node_type": "root" }, { "id": 2, "question": "What is Jason's home country?", "dependancies": [], "node_type": "root" }, { "id": 3, "question": "What is the population of Jason's home country?", "dependancies": [ 2 ], "node_type": "intermediate" }, { "id": 4, "question": "What is the difference in populations of Canada and the Jason's home country?", "dependancies": [ 1, 3 ], "node_type": "end" } ] }

It still seems to be ignoring the description attribute on that field and trying to use "root", "intermediate", "end" - similar to the strings it was passing to the enum previously.

— Reply to this email directly, view it on GitHub https://github.com/jxnl/instructor/issues/77#issuecomment-1789892814, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI254WLCF3QZBDG7KP2QB3LYCLVN7AVCNFSM6AAAAAA3RNKY7KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBZHA4TEOBRGQ . You are receiving this because you were mentioned.Message ID: @.***>

[alexclaydon]

They're tracking an issue over on Marvin that may be related: https://github.com/PrefectHQ/marvin/issues/604#issuecomment-1791676517

[jxnl]

The short term sollution would be to use Literal["one' ,"two"] Or prompt the llm more strictly