Python: GPT4o function calling occasionally buggy?

[bbence84]

Describe the bug Using GPT4 model version 2024-05-13 on Azure (Sweden) with Semantic Kernel, Python. I have noticed that sometimes it seems that GPT4o (os is it in SK?) gets the toolname wrong, e.g. I am getting error:

{'role': 'tool', 'content': 'The tool call with namecc_code.MigrateCodeToCalcis not part of the provided tools, please try again with a supplied tool call name and make sure to validate the name.'

However in fact in the tools list in the JSON I have the function: 'function': {'name': 'cc_code-MigrateCodeToCalc', ...

Is it that GPT4o sometimes puts a dot instead of a dash in the function name?

Interestingly, after such an error, I am getting the below error message too: "Invalid \'messages[5].tool_calls[0].function.name\': string does not match pattern. Expected a string that matches the pattern \'^[a-zA-Z0-9_-]+$\'.", \'type\': \'invalid_request_error\', \'param\': \'messages[5].tool_calls[0].function.name\', \'code\': \'invalid_value\'

Seems that this happens more often with a specific function name, but not 100% of the time, but most of the time.

To Reproduce

Expected behavior Function call works all the time.

Platform Python, SK 1.5.1

[moonbox3]

@bbence84, thanks for reporting. This is model related. We've tried to put some guardrails in place inside of SK to help prevent this type of issue (as you can see, we're telling the model to try again with the correct tool name...). We have heard of this a bit more these days. We'll have another look to see if there are some further improvements we can make to make tool calling more robust.

[moonbox3]

related to #7968

[bbence84]

Thanks! I am not sure anymore, if it's model related, at least now I have switched to GPT4 32K (also on Azure, Sweden), and I am still getting the error randomly. Actually it seems to be slightly different, based on the error message:

openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'messages[5].tool_calls[0].function.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.", 'type': 'invalid_request_error', 'param': 'messages[5].tool_calls[0].function.name', 'code': 'invalid_value'}}

Where does the tool name "messages[5].tool_calls[0].function.name" come from? Maybe something is not getting replaced properly inside SK?

Thanks!

[moonbox3]

Thanks! I am not sure anymore, if it's model related, at least now I have switched to GPT4 32K (also on Azure, Sweden), and I am still getting the error randomly. Actually it seems to be slightly different, based on the error message:

openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'messages[5].tool_calls[0].function.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.", 'type': 'invalid_request_error', 'param': 'messages[5].tool_calls[0].function.name', 'code': 'invalid_value'}}

Where does the tool name "messages[5].tool_calls[0].function.name" come from? Maybe something is not getting replaced properly inside SK?

Thanks!

That's the same error. The model is sending back the function.name like plugin.function instead of plugin-function.

[bbence84]

Is there any update on this? I mean is this an LLM or a framework (SK) issue? It did not seem to happen in the past, so either a recent SK version changed something or Microsoft / OpenAI did something on their side to provoke this error.

[moonbox3]

Is there any update on this? I mean is this an LLM or a framework (SK) issue? It did not seem to happen in the past, so either a recent SK version changed something or Microsoft / OpenAI did something on their side to provoke this error.

This is not entirely an SK issue. It's most likely a model issue where OpenAI is continuing to train with a . delimiter, and so it more often produces a result with something like plugin.function instead of plugin-function. We're still tracking this on our end to make improvements, but it may take a little longer as these improvements should be reflected across all three SK languages.

[yashworlikar]

Facing the same issue in the dotnet version, Was there any reason behind choosing - instead of . as the delimiter?