let users add also custom functions

[ndricca]

Is your feature request related to a problem? Please describe. To achieve a better reasoning planning I would like to pass both functionsand response_model to openai client.chat.completions.create. At the moment response_model will override the provided functions.

Describe the solution you'd like I think it would be sufficient to edit function handle_response_model (script reference instructor/patch.py:81-83) to update new_kwargs instead of overwriting keys"functions" and "function_call".

Describe alternatives you've considered I have tried adding a custom method to my response_model object but methods are not translated to function definitions in OpenAISchema class in script instructor/function_calls.py (neither in pydantic.BaseModel.model_json_schema).

Alternatively, I have to make 2 different invocations (1 with functions and 1 with response model) and choose what to do, instead of letting the LLM choose if the custom function is needed to properly extract an entity (which is my use case).

Additional context Reference packages version:

openai==1.7.0
instructor==0.4.6
pydantic==2.5.3

I am available to change the code as suggested and create a PR, if you give me proper instructions.

Thanks for your work!

[jxnl]

Why not just make it another model? Why make it a function? The point is that the response type is exactly matching the response model.

[ndricca]

Tool calls have been finetuned on lot of python code so I think we should adhere at most to what is usually done, which I think is using data models for entities, not for functions. I feel like incapsulate all to a common BaseModel object and mixing functions and entities could lead to worst results.

I have tried explaining what I am saying by creating two different examples and I think that there are 2 key points:

1. passing functions through a BaseModel will create more verbose argument to tools and this could produce worst results
2. adapting functions to response model will create a not necessary more complicated response model in which I have to explain more in the prompt what to do with this model.

So basically ok, I can incapsulate always all in a BaseModel, but it is really the best option? In the first approach reported below I had to explain more what I have to do, and I have to write more code. The second one seems more clear. I have not commented too much the code below so here's a short summary:

• assume I have bank account transactions to categorize
• sometimes the information in the transaction is enough for openai to understand a category: in that case I will obtain my TransactionCategory entity
• other times a google search is needed: in that case I will pass the google results back as other messages and ask again to assign a category.

Use Case: bank transaction categorization

First Approach: incapsulating both conditional search and transaction categorization to a unique response model

def search_on_google(query: str) -> str:
    """ Search on Google to extract information about a merchant or other elements usuful to help classify a bank transaction. """
    # mock
    print(f"Searching  with query `{query}`")

    result_string = "Google_results:\n "
    if 'ndricca' in query.lower():
        google_results = [
                'NDRICCA is an argentinian company involved in skincare products which sells...',
                'Try CrazyShampoo, the new hair product from Ndricca Industries! Cr...'
            ]
    else:
        google_results = []
    if len(google_results) > 0:
        result_string += "\n ".join([f"{i}. {res}" for i, res in enumerate(google_results)])
    else:
        result_string += "No good results from google"
    return result_string

class Search(BaseModel):
    query: str = Field(
        ...,
        description="Query to search for information about a merchant or other elements usuful to help classify a bank transaction"
    )

    def execute(self) -> str:
        print(f"Searching  with query `{self.query}`")
        google_results = search_on_google(self.query)
        return google_results

class Response(BaseModel):
    chain_of_thought: str = Field(...,
                                  description="Step by step reasoning. If you can assign a category, populate the category attribute. Otherwise use the search attribute to look for results on Google.")
    search: Search | None = None
    category: ExpenseCategory | None = None

def assign_category_or_search(messages: list[dict[str, str]]) -> Response:
    response = client.chat.completions.create(
        model=cst.ConfigOpenAIGPT35Turbo.deployment,
        response_model=Response,
        temperature=0.7,
        messages=messages,
    )
    return response

First case: no need for search

system_message = {
    "role": "system",
    "content": """"The final task is to assign a category to this bank transaction. 
If you are not able to assign, use search to look at Google results and do not provide a category (set to None).
On contrary if you are sure about a category, set search argument as None.
""",
}
user_message_1 = {
    "role": "user",
    "content": "Order ID-XXXXX MacDonald's DD/MM/YYYY Rif. Buenos Aires",
}

res_1 = assign_category_or_search(messages=[system_message, user_message_1])
print(res_1.model_dump_json(indent=4))
# {
#     "chain_of_thought": "The transaction is from MacDonald's and the reference mentions Buenos Aires. This is likely a food expense related to MacDonald's in Buenos Aires.",
#     "search": null,
#     "category": {
#         "chain_of_thought": "This is a food expense at MacDonald's in Buenos Aires.",
#         "merchant_name": "MacDonald's",
#         "espense_type": "Food",
#         "detail": "Food expense at MacDonald's in Buenos Aires."
#     }
# }

Second case: search is needed

user_message_2 = {
    "role": "user",
    "content": "Order ID-XXXXX Ndricca Buenos Aires DD/MM/YYYY Rif. Buenos Aires",
}
res_2 = assign_category_or_search(messages=[system_message, user_message_2])
print(res_2.model_dump_json(indent=4))
# {
#     "chain_of_thought": "The transaction appears to be related to an order from Ndricca in Buenos Aires. The reference 'Rif. Buenos Aires' suggests that it may be a payment for an order or a delivery from Ndricca.",
#     "search": {
#         "query": "Ndricca Buenos Aires"
#     },
#     "category": null
# }
if not res_2.category and res_2.search:
    assistant_message = {
        "role": "assistant",
        "tool_calls": [res_2._raw_response.choices[0].message.tool_calls[0].model_dump()]
    }
    query_results = res_2.search.execute()
    print(query_results)
    tool_message = {
        "role": "tool",
        "tool_call_id": res_2._raw_response.choices[0].message.tool_calls[0].id,
        "content": query_results
    }
    user_message_reinforced = {
        "role": "user",
        "content": f"Now given the additional information try to assign a category to the transaction: \n{user_message_2['content']}"
    }
    second_res = assign_category_or_search(
        messages=[system_message, user_message_2, assistant_message, tool_message, user_message_reinforced]
    )
    print(second_res.model_dump_json(indent=4))
    # {
    #     "chain_of_thought": "The transaction is likely related to a purchase of skincare products from Ndricca in Buenos Aires. Therefore, the category is likely 'Health'.",
    #     "search": null,
    #     "category": {
    #         "chain_of_thought": "The transaction is related to a purchase of skincare products from Ndricca in Buenos Aires.",
    #         "merchant_name": "Ndricca",
    #         "espense_type": "Health",
    #         "detail": "Skincare products purchase"
    #     }
    # }

Second Approach: separating conditional search from categorization response model

from instructor.function_calls import openai_schema

tools = [
    {

        "type": "function",
        "function": openai_schema(ExpenseCategory).openai_schema
    },
    {
        "type": "function",
        "function": {
            "name": search_on_google.__name__,
            "description": search_on_google.__doc__,
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "Query to obtain useful results.",
                    },
                },
                "required": ["object"],
            },
        }}
]

def assign_category_or_search_other(messages: list[dict[str, str]]):
    response = client.chat.completions.create(
        model=cst.ConfigOpenAIGPT35Turbo.deployment,
        tools=tools,
        temperature=0.7,
        messages=messages,
    )
    return response

First case: no need for search

system_message_other = {
    "role": "system",
    "content": """"The final task is to assign a category to this bank transaction. 
If you are not able to assign, search for more information using Google and do not provide a category.
On contrary if you are sure about a category, assign it without searching.
""",
}
user_message_1 = {
    "role": "user",
    "content": "Order ID-XXXXX MacDonald's DD/MM/YYYY Rif. Buenos Aires",
}

res_1_other = assign_category_or_search_other(messages=[system_message_other, user_message_1])
print(res_1_other.choices[0].message.tool_calls[0].model_dump_json(indent=4))
# {
#     "id": "call_qLxTCPthAK5zbNki2EX9CtI5",
#     "function": {
#         "arguments": "{\"chain_of_thought\":\"This is a transaction at MacDonald's, which is a fast-food restaurant. The transaction seems to be related to food and dining out, so it should be categorized as 'Food'.\",\"merchant_name\":\"MacDonald's\",\"espense_type\":\"Food\",\"detail\":\"Order ID-XXXXX MacDonald's DD/MM/YYYY Rif. Buenos Aires\"}",
#         "name": "ExpenseCategory"
#     },
#     "type": "function"
# }

Second case: search is needed

user_message_2 = {
    "role": "user",
    "content": "Order ID-XXXXX Ndricca Buenos Aires DD/MM/YYYY Rif. Buenos Aires",
}
res_2_other = assign_category_or_search_other(messages=[system_message_other, user_message_2])
print(res_2_other.choices[0].message.tool_calls[0].model_dump_json(indent=4))
# {
#     "id": "call_gWMjuNmLmNOiO0s2xIdGrC9q",
#     "function": {
#         "arguments": "{\"query\": \"Ndricca Buenos Aires\"}",
#         "name": "search_on_google"
#     },
#     "type": "function"
# }
if res_2_other.choices[0].message.tool_calls[0].function.name == "search_on_google":
    assistant_message = {
        "role": "assistant",
        "tool_calls": [res_2_other.choices[0].message.tool_calls[0].model_dump()]
    }
    query_results = search_on_google(**json.loads(res_2_other.choices[0].message.tool_calls[0].function.arguments))
    print(query_results)
    tool_message = {
        "role": "tool",
        "tool_call_id": res_2_other.choices[0].message.tool_calls[0].id,
        "content": query_results
    }
    user_message_reinforced = {
        "role": "user",
        "content": f"Now given the additional information try to assign a category to the transaction: \n{user_message_2['content']}"
    }
    second_res_other = assign_category_or_search_other(
        messages=[system_message, user_message_2, assistant_message, tool_message, user_message_reinforced]
    )
    print(second_res_other.choices[0].message.tool_calls[0].model_dump_json(indent=4))
    # {
    #     "id": "call_GdQleGdZXwiBXkHFxrcBk39F",
    #     "function": {
    #         "arguments": "{\"chain_of_thought\":\"The transaction seems to be related to an order from Ndricca Buenos Aires, which is an Argentinian company involved in skincare products. The order ID is mentioned, indicating a purchase or an order. The reference 'Rif. Buenos Aires' also suggests that this is a purchase from Buenos Aires. Considering the nature of the merchant and the reference to an order, it is likely that this transaction falls under the 'Health' expense category.\",\"merchant_name\":\"Ndricca Buenos Aires\",\"espense_type\":\"Health\",\"detail\":\"Purchase or Order from Ndricca Buenos Aires\"}",
    #         "name": "ExpenseCategory"
    #     },
    #     "type": "function"
    # }

[jxnl]

https://jxnl.github.io/instructor/concepts/parallel/

[ndricca]

I understood that this is a "not planned" request because it would be against the general design of the package