Python: FunctionCallingStepwisePlanner results in "Function Call arguments are not valid JSON.. "

[hchoi8782]

Describe the bug When running plug-ins w/ the FunctionCallingStepwisePlanner, many of the tool calls results in an invalid JSON error. After repeated invalid JSON errors, it fails w/ Error 400

To Reproduce Here's the plan "ask" (the {{$...}} are replaced by my code at runtime: Complete the following tasks for each of the following SNR investigation data .csv files one by one: {{$INVESTIGATION_DATA_PATHS}}

1. Read and analyze the SNR investigation data in the .csv file.
2. Generate a suitable filename for the analsys based on the content.
3. Output the analysis to blob storage using the generated filename in the following directory: '{{$SNR_DIRECTORY_PATH}}/hermes' After the summaries have all been saved to blob storage, respond with 'exit' or 'TERMINATE'.

Expected behavior There are several plug-ins that do the steps requested. The plug-ins should be called w/ the appropriate parameters.

Screenshots INFO:semantic_kernel.kernel:Received invalid arguments for function core_prompts-generate_filename_from_content: Function Call arguments are not valid JSON.. Trying tool call again.

INFO:semantic_kernel.kernel:Received invalid arguments for function multi_tool_use.parallel: Function Call arguments are not valid JSON.. Trying tool call again.
INFO:httpx:HTTP Request: POST https://stg-mc-ncus-openai-api.openai.azure.com/openai/deployments/gpt-4o-adhoc/chat/completions?api-version=2024-02-01 "HTTP/1.1 400 model_error"

---------------------------------------------------------------------------
BadRequestError                           Traceback (most recent call last)
File /usr/local/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/open_ai_handler.py:40, in OpenAIHandler._send_request(self, request_settings)
     39 if self.ai_model_type == OpenAIModelTypes.CHAT:
---> 40     response = await self.client.chat.completions.create(**request_settings.prepare_settings_dict())
     41 else:

File /usr/local/lib/python3.12/site-packages/openai/resources/chat/completions.py:1295, in AsyncCompletions.create(self, messages, model, frequency_penalty, function_call, functions, logit_bias, logprobs, max_tokens, n, parallel_tool_calls, presence_penalty, response_format, seed, service_tier, stop, stream, stream_options, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)
   1261 @required_args(["messages", "model"], ["messages", "model", "stream"])
   1262 async def create(
   1263     self,
   (...)
   1293     timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN,
   1294 ) -> ChatCompletion | AsyncStream[ChatCompletionChunk]:
-> 1295     return await self._post(
   1296         "/chat/completions",
   1297         body=await async_maybe_transform(
   1298             {
   1299                 "messages": messages,
   1300                 "model": model,
   1301                 "frequency_penalty": frequency_penalty,
   1302                 "function_call": function_call,
   1303                 "functions": functions,
   1304                 "logit_bias": logit_bias,
   1305                 "logprobs": logprobs,
   1306                 "max_tokens": max_tokens,
   1307                 "n": n,
   1308                 "parallel_tool_calls": parallel_tool_calls,
   1309                 "presence_penalty": presence_penalty,
   1310                 "response_format": response_format,
   1311                 "seed": seed,
   1312                 "service_tier": service_tier,
   1313                 "stop": stop,
   1314                 "stream": stream,
   1315                 "stream_options": stream_options,
   1316                 "temperature": temperature,
   1317                 "tool_choice": tool_choice,
   1318                 "tools": tools,
   1319                 "top_logprobs": top_logprobs,
   1320                 "top_p": top_p,
   1321                 "user": user,
   1322             },
   1323             completion_create_params.CompletionCreateParams,
   1324         ),
   1325         options=make_request_options(
   1326             extra_headers=extra_headers, extra_query=extra_query, extra_body=extra_body, timeout=timeout
   1327         ),
   1328         cast_to=ChatCompletion,
   1329         stream=stream or False,
   1330         stream_cls=AsyncStream[ChatCompletionChunk],
   1331     )

File /usr/local/lib/python3.12/site-packages/openai/_base_client.py:1836, in AsyncAPIClient.post(self, path, cast_to, body, files, options, stream, stream_cls)
   1833 opts = FinalRequestOptions.construct(
   1834     method="post", url=path, json_data=body, files=await async_to_httpx_files(files), **options
   1835 )
-> 1836 return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)

File /usr/local/lib/python3.12/site-packages/openai/_base_client.py:1524, in AsyncAPIClient.request(self, cast_to, options, stream, stream_cls, remaining_retries)
   1515 async def request(
   1516     self,
   1517     cast_to: Type[ResponseT],
   (...)
   1522     remaining_retries: Optional[int] = None,
   1523 ) -> ResponseT | _AsyncStreamT:
-> 1524     return await self._request(
   1525         cast_to=cast_to,
   1526         options=options,
   1527         stream=stream,
   1528         stream_cls=stream_cls,
   1529         remaining_retries=remaining_retries,
   1530     )

File /usr/local/lib/python3.12/site-packages/openai/_base_client.py:1625, in AsyncAPIClient._request(self, cast_to, options, stream, stream_cls, remaining_retries)
   1624     log.debug("Re-raising status error")
-> 1625     raise self._make_status_error_from_response(err.response) from None
   1627 return await self._process_response(
   1628     cast_to=cast_to,
   1629     options=options,
   (...)
   1633     retries_taken=options.get_max_retries(self.max_retries) - retries,
   1634 )

BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'messages[47].tool_calls[0].function.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.", 'type': 'invalid_request_error', 'param': 'messages[47].tool_calls[0].function.name', 'code': 'invalid_value'}}

The above exception was the direct cause of the following exception:

ServiceResponseException                  Traceback (most recent call last)
Cell In[3], line 38
      7 playbook["plays"] = [
      8     {
      9     "kernel": "semantic-kernel",
   (...)
     33 }
     34 ]
     36 print(json.dumps(playbook, indent=2))
---> 38 result = await engine.execute_playbook(playbook)
     39 print(result)
     40 engine.evaluator.get_results()

File /project/engine/CognitionEngine.py:155, in CognitionEngine.execute_playbook(self, playbook)
    153 log.info(f"Executing PLAY #{i}...")
    154 # play execution
--> 155 result = await self.execute_play(play.copy(), play_input)
    157 # "feed forward" the result into the next play
    158 play_input = str(result)

File /project/engine/CognitionEngine.py:111, in CognitionEngine.execute_play(self, play, play_input)
    109 # execute the play through the kernel
    110 if asyncio.iscoroutinefunction(self._adapters["kernel"].execute_play):
--> 111     result = await self._adapters["kernel"].execute_play(ask, play, play_input)
    112 else:
    113     result = self._adapters["kernel"].execute_play(ask, play, play_input)

File /project/engine/kernel/SemanticKernel.py:150, in SemanticKernel.execute_play(self, ask, play, play_input)
    148 # NOTE: the 'result' is not a string, it is a KernelContext object --> this may have change in v0.9
    149 if planner == "stepwise":
--> 150     result = await plan.invoke(self.kernel, question=ask, arguments=kernel_args)
    151 else:
    152     result = await plan.invoke(self.kernel, arguments=kernel_args)

File /usr/local/lib/python3.12/site-packages/semantic_kernel/planners/function_calling_stepwise_planner/function_calling_stepwise_planner.py:166, in FunctionCallingStepwisePlanner.invoke(self, kernel, question, arguments, **kwargs)
    164 # For each step, request another completion to select a function for that step
    165 chat_history_for_steps.add_user_message(STEPWISE_USER_MESSAGE)
--> 166 chat_result = await chat_completion.get_chat_message_contents(
    167     chat_history=chat_history_for_steps,
    168     settings=prompt_execution_settings,
    169     kernel=cloned_kernel,
    170 )
    171 chat_result = chat_result[0]
    172 chat_history_for_steps.add_message(chat_result)

File /usr/local/lib/python3.12/site-packages/semantic_kernel/utils/telemetry/decorators.py:87, in trace_chat_completion.<locals>.inner_trace_chat_completion.<locals>.wrapper_decorator(*args, **kwargs)
     82 span = _start_completion_activity(
     83     CHAT_COMPLETION_OPERATION, model_name, model_provider, formatted_messages, settings
     84 )
     86 try:
---> 87     completions: list[ChatMessageContent] = await completion_func(*args, **kwargs)
     88 except Exception as exception:
     89     if span:

File /usr/local/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py:105, in OpenAIChatCompletionBase.get_chat_message_contents(self, chat_history, settings, **kwargs)
    101 self._prepare_settings(settings, chat_history, stream_request=False, kernel=kernel)
    102 if settings.function_choice_behavior is None or (
    103     settings.function_choice_behavior and not settings.function_choice_behavior.auto_invoke_kernel_functions
    104 ):
--> 105     return await self._send_chat_request(settings)
    107 # loop for auto-invoke function calls
    108 for request_index in range(settings.function_choice_behavior.maximum_auto_invoke_attempts):

File /usr/local/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py:247, in OpenAIChatCompletionBase._send_chat_request(self, settings)
    245 async def _send_chat_request(self, settings: OpenAIChatPromptExecutionSettings) -> list["ChatMessageContent"]:
    246     """Send the chat request."""
--> 247     response = await self._send_request(request_settings=settings)
    248     assert isinstance(response, ChatCompletion)  # nosec
    249     response_metadata = self._get_metadata_from_chat_response(response)

File /usr/local/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/open_ai_handler.py:51, in OpenAIHandler._send_request(self, request_settings)
     46     if ex.code == "content_filter":
     47         raise ContentFilterAIException(
     48             f"{type(self)} service encountered a content error",
     49             ex,
     50         ) from ex
---> 51     raise ServiceResponseException(
     52         f"{type(self)} service failed to complete the prompt",
     53         ex,
     54     ) from ex
     55 except Exception as ex:
     56     raise ServiceResponseException(
     57         f"{type(self)} service failed to complete the prompt",
     58         ex,
     59     ) from ex

ServiceResponseException: ("<class 'semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion.AzureChatCompletion'> service failed to complete the prompt", BadRequestError('Error code: 400 - {\'error\': {\'message\': "Invalid \'messages[47].tool_calls[0].function.name\': string does not match pattern. Expected a string that matches the pattern \'^[a-zA-Z0-9_-]+$\'.", \'type\': \'invalid_request_error\', \'param\': \'messages[47].tool_calls[0].function.name\', \'code\': \'invalid_value\'}}'))

Platform

• OS: Linux
• IDE: Jupyter running inside a Docker container
• Language: Python
• Source: v1.4.0

Additional context Add any other context about the problem here.

[moonbox3]

The real issue here is that the model is potentially sending back a function name that doesn't fit our plugin/function name regex criteria:

Error code: 400 - {\'error\': {\'message\': "Invalid \'messages[47].toolcalls[0].function.name\': string does not match pattern. Expected a string that matches the pattern \'^[a-zA-Z0-9-]+$\'.", \'type\': \'invalid_request_error\', \'param\': \'messages[47].tool_calls[0].function.name\', \'code\': \'invalid_value\'}}'))

We should look at the function name we get and if we get one that fails the regex, we should tell the model how the function name should be formed (similar to how we tell the model the JSON args are invalid).