trypear / pearai-server-issues-public

Repository dedicated to create issues for PearAI server.
6 stars 0 forks source link

Handle AI Model Exceptions [Anthropic, OpenAI, etc] #5

Open ItWasEnder opened 2 months ago

ItWasEnder commented 2 months ago

There is a risk of the AI SDKs generating unhandled exceptions, such as 429 and overload exceptions. To prevent fatal crashes, we need to implement handling these errors in a more graceful way.

Example

[pearai-server-api2] [2024-09-01 21:03:53] [2024-09-01 21:03:53 +0000] [19] [ERROR] Error handling request
[pearai-server-api2] [2024-09-01 21:03:53] Traceback (most recent call last):
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 183, in handle_request
[pearai-server-api2] [2024-09-01 21:03:53]     for item in respiter:
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/werkzeug/wsgi.py", line 256, in __next__
[pearai-server-api2] [2024-09-01 21:03:53]     return self._next()
[pearai-server-api2] [2024-09-01 21:03:53]            ^^^^^^^^^^^^
[pearai-server-api2] [2024-09-01 21:17:25]     return self._request(
[pearai-server-api2] [2024-09-01 21:17:25]            ^^^^^^^^^^^^^^
[pearai-server-api2] [2024-09-01 21:17:25]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/anthropic/_base_client.py", line 1040, in _request
[pearai-server-api2] [2024-09-01 21:17:25]     raise self._make_status_error_from_response(err.response) from None
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded
[pearai-server-api2] [2024-09-01 21:03:53]     for item in iterable:
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/flask/helpers.py", line 113, in generator
[pearai-server-api2] [2024-09-01 21:03:53]     yield from gen
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/app/services/llm/anthropic_service.py", line 61, in generate
[pearai-server-api2] [2024-09-01 21:03:53]     for chunk in response:
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/anthropic/_streaming.py", line 68, in __iter__
[pearai-server-api2] [2024-09-01 21:03:53]     for item in self._iterator:
[pearai-server-api2] [2024-09-01 21:03:53]   File "/workspace/api/.heroku/python/lib/python3.12/site-packages/anthropic/_streaming.py", line 110, in __stream__
[pearai-server-api2] [2024-09-01 21:03:53]     raise self._client._make_status_error(
[pearai-server-api2] [2024-09-01 21:03:53] anthropic.APIStatusError: {'type': 'error', 'error': {'details': None, 'type': 'overloaded_error', 'message': 'Overloaded'}}
nang-dev commented 2 months ago

we have model swtiching

ItWasEnder commented 2 months ago

@nang-dev Rewrote. We should be handling the exceptions in some way instead of letting them just be caught by flask.

nang-dev commented 2 months ago

tru

cqdev-co commented 2 months ago

I got this

nang-dev commented 2 months ago

goat @cqdev-co

cqdev-co commented 2 months ago

Done and merged

Source: trust me bro