Load failed in TypingMind and big-AGI

[vlbosch]

I tried to run some models with Toolio in server mode. I'm running the server in DEBUG to see all the logs. When prompting a model in both TypingMind and big-AGI, I get "Load failed" instead of the output of the model. However, in the debug logs I can see the model replied or even called a function.

In TypingMind, when turning off streaming, it correctly shows that it's waiting for a response. As soon as the response should come in, it reports the Load Failed error. When I enable streaming, it immediately throws an error: "ERROR: Exception in ASGI application Traceback (most recent call last): File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/responses.py", line 257, in call await wrap(partial(self.listen_for_disconnect, receive)) File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/responses.py", line 253, in wrap await func() File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/responses.py", line 230, in listen_for_disconnect message = await receive() ^^^^^^^^^^^^^^^ File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 534, in receive await self.message_event.wait() File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/asyncio/locks.py", line 212, in wait await fut asyncio.exceptions.CancelledError: Cancelled by cancel scope ba497f530

During handling of the above exception, another exception occurred:

• Exception Group Traceback (most recent call last): | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi | result = await app( # type: ignore[func-returns-value] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call | return await self.app(scope, receive, send) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call | await super().call(scope, receive, send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/applications.py", line 113, in call | await self.middleware_stack(scope, receive, send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call | raise exc | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call | await self.app(scope, receive, _send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call | await self.simple_response(scope, receive, send, request_headers=headers) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response | await self.app(scope, receive, send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app | raise exc | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app | await app(scope, receive, sender) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/routing.py", line 715, in call | await self.middleware_stack(scope, receive, send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/routing.py", line 735, in app | await route.handle(scope, receive, send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle | await self.app(scope, receive, send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/routing.py", line 76, in app | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app | raise exc | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app | await app(scope, receive, sender) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/routing.py", line 74, in app | await response(scope, receive, send) | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/responses.py", line 250, in call | async with anyio.create_task_group() as task_group: | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 736, in aexit | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/responses.py", line 253, in wrap | await func() | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/starlette/responses.py", line 242, in stream_response | async for chunk in self.body_iterator: | File "/opt/homebrew/Caskroom/miniconda/base/envs/toolio/lib/python3.12/site-packages/toolio/http_impl.py", line 57, in post_v1_chat_completions_impl | responder = ToolCallStreamingResponder(model_name, model_type) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | TypeError: ToolCallStreamingResponder.init() missing 1 required positional argument: 'tools' +------------------------------------ "

Please let me know if I can do anything to help triage this issue.

[vlbosch]

In addition, big-AGI reports the following error: "[Service Issue] Openai: fetch failed - SocketError: other side closed · {"name":"SocketError","code":"UND_ERR_SOCKET","socket":"

[uogbuji]

Hi, I'm familiar with neither TypingMind nor big-AGI. Are you able to reproduce this error with a simple cURL, by any chance?

If not, can you provide minimal but complete instructions to reproduce? Need to know what to install (and how, depending on if it's just a simple pip requirement). Then what, in detail, is done to trigger the traceback?

Also, are you using Toolio 0.5.0? Have you ttried installing the main branch? There have been some fixes since the last release.