OpenAI: Failure on model function name hallucination

[JoongWonSeo]

There are currently several known failure cases/limitations for GPT function calling:

• Parallel Calls: Model makes a syntax mistake and the OpenAI-internal parallel calling syntax "leaks" into a normal function call
• The streaming for parallel calls only happens at the very end once the full parallel call has been generated, and then streams all at once (probably they don't know how to parse auto-complete like we do)
• If the model hallucinates a function name like "assistant.web.tool", any subsequent api call will be rejected as 400 bad request, because the function tool name may not contain a dot. The wrapping function needs to detect these and automatically filter out and then retry or fail.

[JoongWonSeo]

see:

copilot-1   | 2024-06-02 19:25:27 - httpcore.connection - DEBUG - connect_tcp.started host='api.openai.com' port=443 local_address=None timeout=5.0 socket_options=None
copilot-1   | 2024-06-02 19:25:27 - httpcore.connection - DEBUG - connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0xffff63d24150>
copilot-1   | 2024-06-02 19:25:27 - httpcore.connection - DEBUG - start_tls.started ssl_context=<ssl.SSLContext object at 0xffff63d56570> server_hostname='api.openai.com' timeout=5.0
copilot-1   | 2024-06-02 19:25:27 - httpcore.connection - DEBUG - start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0xffff63d27f10>
copilot-1   | 2024-06-02 19:25:27 - httpcore.http11 - DEBUG - send_request_headers.started request=<Request [b'POST']>
copilot-1   | 2024-06-02 19:25:27 - httpcore.http11 - DEBUG - send_request_headers.complete
copilot-1   | 2024-06-02 19:25:27 - httpcore.http11 - DEBUG - send_request_body.started request=<Request [b'POST']>
copilot-1   | 2024-06-02 19:25:27 - httpcore.http11 - DEBUG - send_request_body.complete
copilot-1   | 2024-06-02 19:25:27 - httpcore.http11 - DEBUG - receive_response_headers.started request=<Request [b'POST']>
copilot-1   | 2024-06-02 19:25:28 - httpcore.http11 - DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 400, b'Bad Request', [(b'Date', b'Sun, 02 Jun 2024 19:25:28 GMT'), (b'Content-Type', b'application/json'), (b'Content-Length', b'304'), (b'Connection', b'keep-alive'), (b'openai-organization', b'user-s0dtvdy3yyxx9o0vusmtny08'), (b'openai-processing-ms', b'68'), (b'openai-version', b'2020-10-01'), (b'strict-transport-security', b'max-age=15724800; includeSubDomains'), (b'x-ratelimit-limit-requests', b'10000'), (b'x-ratelimit-limit-tokens', b'800000'), (b'x-ratelimit-remaining-requests', b'9999'), (b'x-ratelimit-remaining-tokens', b'797590'), (b'x-ratelimit-reset-requests', b'6ms'), (b'x-ratelimit-reset-tokens', b'180ms'), (b'x-request-id', b'req_04cb1c8dc4a445985454cc60a0ca56a4'), (b'CF-Cache-Status', b'DYNAMIC'), (b'Set-Cookie', b'__cf_bm=eMoP8f6xx1h0XYkLKyT8txOvLNPX.FJ_a2kIWxOPgmw-1717356328-1.0.1.1-ELQXWuQKuD.kDYc3GwsgsVbi2XsnVj1FYdNrTNMiu3OuIsjANix3J4DO58iZVirVFBzZ7ny07GfmmsOdRCZIVQ; path=/; expires=Sun, 02-Jun-24 19:55:28 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None'), (b'Set-Cookie', b'_cfuvid=bcLuiN6_GvSLiYADPjAmo.r91gTr4xhYOvoZaB5Wx3s-1717356328261-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None'), (b'Server', b'cloudflare'), (b'CF-RAY', b'88d9d4583b163654-FRA'), (b'alt-svc', b'h3=":443"; ma=86400')])
copilot-1   | 2024-06-02 19:25:28 - httpx - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 400 Bad Request"
copilot-1   | 2024-06-02 19:25:28 - openai._base_client - DEBUG - HTTP Request: POST https://api.openai.com/v1/chat/completions "400 Bad Request"
copilot-1   | 2024-06-02 19:25:28 - openai._base_client - DEBUG - Encountered httpx.HTTPStatusError
copilot-1   | Traceback (most recent call last):
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 1555, in _request
copilot-1   |     response.raise_for_status()
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/httpx/_models.py", line 761, in raise_for_status
copilot-1   |     raise HTTPStatusError(message, request=request, response=self)
copilot-1   | httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.openai.com/v1/chat/completions'
copilot-1   | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
copilot-1   | 2024-06-02 19:25:28 - openai._base_client - DEBUG - Not retrying
copilot-1   | 2024-06-02 19:25:28 - httpcore.http11 - DEBUG - receive_response_body.started request=<Request [b'POST']>
copilot-1   | 2024-06-02 19:25:28 - httpcore.http11 - DEBUG - receive_response_body.complete
copilot-1   | 2024-06-02 19:25:28 - httpcore.http11 - DEBUG - response_closed.started
copilot-1   | 2024-06-02 19:25:28 - httpcore.http11 - DEBUG - response_closed.complete
copilot-1   | 2024-06-02 19:25:28 - openai._base_client - DEBUG - Re-raising status error
copilot-1   | 2024-06-02 19:25:28 - asyncio - ERROR - Task exception was never retrieved
copilot-1   | future: <Task finished name='Task-5973' coro=<Sync._create_task_handlers.<locals>._run_and_pop() done, defined at /app/backend/.venv/lib/python3.11/site-packages/ws_sync/sync.py:435> exception=BadRequestError('Error code: 400 - {\'error\': {\'message\': "Invalid \'messages[4].tool_calls[0].function.name\': string does not match pattern. Expected a string that matches the pattern \'^[a-zA-Z0-9_-]+$\'.", \'type\': \'invalid_request_error\', \'param\': \'messages[4].tool_calls[0].function.name\', \'code\': \'invalid_value\'}}')>
copilot-1   | Traceback (most recent call last):
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/ws_sync/sync.py", line 437, in _run_and_pop
copilot-1   |     await task
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/ws_sync/decorators.py", line 238, in wrapper
copilot-1   |     return await func(self, *args, **kwargs)
copilot-1   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
copilot-1   |   File "/app/backend/uc_copilot/sync/assistant.py", line 72, in prompt
copilot-1   |     async for event in self.response_events(
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/agentools/assistants/chatgpt.py", line 109, in response_events
copilot-1   |     async for partial_event in completion_events:
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/agentools/assistants/chatgpt.py", line 164, in completion_events
copilot-1   |     completion_stream = await openai_chat(**openai_args)
copilot-1   |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/agentools/api/openai.py", line 65, in openai_chat
copilot-1   |     gen = await client.chat.completions.create(**openai_kwargs)
copilot-1   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 1159, in create
copilot-1   |     return await self._post(
copilot-1   |            ^^^^^^^^^^^^^^^^^
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 1782, in post
copilot-1   |     return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
copilot-1   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 1485, in request
copilot-1   |     return await self._request(
copilot-1   |            ^^^^^^^^^^^^^^^^^^^^
copilot-1   |   File "/app/backend/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 1576, in _request
copilot-1   |     raise self._make_status_error_from_response(err.response) from None
copilot-1   | openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'messages[4].tool_calls[0].function.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.", 'type': 'invalid_request_error', 'param': 'messages[4].tool_calls[0].function.name', 'code': 'invalid_value'}}