bug: Supabase client connection attempts failed

Environment

OS and Architecture: RKE2 (airgapped)
App or Package Name: LeapfrogAI
App or Package Version: v0.11.0
Kubernetes Distribution: RKE2
Kubernetes Version: v1.28.2
Other:

Steps to reproduce

Set up 10+ users with LFAI
Add multiple assistants and a large number of files to the file management system (varied number of assistants and files per user).
Have the users logged in at the same time and have the users perform various operations.

Expected result

We would expect supabase to be able to handle the simultaneous connections. Would not expect the API and UI to encounter issues given the number of users and files uploaded.

Actual Result

Various users were kicked out of the session / unable to perform application operations. User requests were rejected while seeing the supabase errors.
Restarting the leapfrogai-api deployment was the workaround for being able to fix the issue

Visual Proof (screenshots, videos, text, etc)

LeapfrogAI API pod logs:


File "/leapfrogai/.venv/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/leapfrogai/.venv/lib/python3.11/site-packages/httpcore/_async/connection.py", line 122, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/leapfrogai/.venv/lib/python3.11/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp
return await self._backend.connect_tcp(
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/leapfrogai/.venv/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
with map_exceptions(exc_map):
File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/leapfrogai/.venv/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/leapfrogai/.venv/lib/python3.11/site-packages/gotrue/_async/gotrue_base_api.py", line 111, in _request response = await self._http_client.request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/leapfrogai/.venv/lib/python3.11/site-packages/httpx/_client.py", line 1574, in request return await self.send(request, auth=auth, follow_redirects=follow_redirects) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/leapfrogai/.venv/lib/python3.11/site-packages/httpx/_client.py", line 1661, in send response = await self._send_handling_auth( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/leapfrogai/.venv/lib/python3.11/site-packages/httpx/_client.py", line 1689, in _send_handling_auth response = await self._send_handling_redirects( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/leapfrogai/.venv/lib/python3.11/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects response = await self._send_single_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/leapfrogai/.venv/lib/python3.11/site-packages/httpx/_client.py", line 1763, in _send_single_request response = await transport.handle_async_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/leapfrogai/.venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 372, in handle_async_request with map_httpcore_exceptions(): File "/usr/lib/python3.11/contextlib.py", line 155, in exit self.gen.throw(typ, value, traceback) File "/leapfrogai/.venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ConnectError: All connection attempts failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/leapfrogai/.venv/lib/python3.11/site-packages/leapfrogai_api/routers/supabase_session.py", line 60, in init_supabase_client await client.auth.set_session( File "/leapfrogai/.venv/lib/python3.11/site-packages/gotrue/_async/gotrue_client.py", line 640, in set_session response = await self.get_user(access_token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/leapfrogai/.venv/lib/python3.11/site-packages/gotrue/_async/gotrue_client.py", line 586, in get_user return await self._request("GET", "user", jwt=jwt, xform=parse_user_response) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/leapfrogai/.venv/lib/python3.11/site-packages/gotrue/_async/gotrue_base_api.py", line 123, in _request raise handle_exception(e) gotrue.errors.AuthRetryableError: All connection attempts failed


- Error message found in leapfrogai-ui logs:

Listening on 0.0.0.0:3000 InternalServerError: 500 status code (no body) at APIError.generate (file:///app/node_modules/openai/error.mjs:62:20) at OpenAI.makeStatusError (file:///app/node_modules/openai/core.mjs:268:25) at OpenAI.makeRequest (file:///app/node_modules/openai/core.mjs:311:30) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async GET (file:///app/build/server/chunks/_server.ts-Bh-eBlsA.js:11:16) at async render_endpoint (file:///app/build/server/index.js:1228:20) at async resolve2 (file:///app/build/server/index.js:3708:22) at async csp (file:///app/build/server/chunks/hooks.server-BtaMPuC-.js:92:20) at async respond (file:///app/build/server/index.js:3603:22) at async Array.ssr (file:///app/build/handler.js:1261:3) { status: 500, headers: { 'content-length': '46', 'content-type': 'application/json', date: 'Tue, 27 Aug 2024 19:32:31 GMT', server: 'envoy', 'x-envoy-upstream-service-time': '33' }, request_id: undefined, error: undefined, code: undefined, param: undefined, type: undefined } Error fetching assistants InternalServerError: 500 status code (no body) at APIError.generate (file:///app/node_modules/openai/error.mjs:62:20) at OpenAI.makeStatusError (file:///app/node_modules/openai/core.mjs:268:25) at OpenAI.makeRequest (file:///app/node_modules/openai/core.mjs:311:30) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async GET (file:///app/build/server/chunks/_server.ts-DJXll3aH.js:12:28) at async render_endpoint (file:///app/build/server/index.js:1228:20) at async resolve2 (file:///app/build/server/index.js:3708:22) at async csp (file:///app/build/server/chunks/hooks.server-BtaMPuC-.js:92:20) at async respond (file:///app/build/server/index.js:3603:22) at async Array.ssr (file:///app/build/handler.js:1261:3) { status: 500, headers: { 'content-length': '46', 'content-type': 'application/json', date: 'Tue, 27 Aug 2024 19:32:31 GMT', server: 'envoy', 'x-envoy-upstream-service-time': '31' }, request_id: undefined, error: undefined, code: undefined, param: undefined, type: undefined } (node:1) ExperimentalWarning: buffer.File is an experimental feature and might change at any time (Use node --trace-warnings ... to show where the warning was created) Error fetching thread or messages: Error: 500 status code (no body) Error fetching assistants InternalServerError: 500 status code (no body) at APIError.generate (file:///app/node_modules/openai/error.mjs:62:20) at OpenAI.makeStatusError (file:///app/node_modules/openai/core.mjs:268:25) at OpenAI.makeRequest (file:///app/node_modules/openai/core.mjs:311:30) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async GET (file:///app/build/server/chunks/_server.ts-DJXll3aH.js:12:28) at async render_endpoint (file:///app/build/server/index.js:1228:20) at async resolve2 (file:///app/build/server/index.js:3708:22) at async csp (file:///app/build/server/chunks/hooks.server-BtaMPuC-.js:92:20) at async respond (file:///app/build/server/index.js:3603:22) at async fetch (file:///app/build/server/index.js:3360:26) { status: 500, headers: { 'content-length': '46', 'content-type': 'application/json', date: 'Tue, 27 Aug 2024 19:32:45 GMT', server: 'envoy', 'x-envoy-upstream-service-time': '42' }, request_id: undefined, error: undefined, code: undefined, param: undefined, type: undefined } Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) InternalServerError: 500 status code (no body) at APIError.generate (file:///app/node_modules/openai/error.mjs:62:20) at OpenAI.makeStatusError (file:///app/node_modules/openai/core.mjs:268:25) at OpenAI.makeRequest (file:///app/node_modules/openai/core.mjs:311:30) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async GET (file:///app/build/server/chunks/_server.ts-Bh-eBlsA.js:11:16) at async render_endpoint (file:///app/build/server/index.js:1228:20) at async resolve2 (file:///app/build/server/index.js:3708:22) at async csp (file:///app/build/server/chunks/hooks.server-BtaMPuC-.js:92:20) at async respond (file:///app/build/server/index.js:3603:22) at async fetch (file:///app/build/server/index.js:3360:26) { status: 500, headers: { 'content-length': '46', 'content-type': 'application/json', date: 'Tue, 27 Aug 2024 19:32:45 GMT', server: 'envoy', 'x-envoy-upstream-service-time': '43' }, request_id: undefined, error: undefined, code: undefined, param: undefined, type: undefined } Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) Error fetching thread or messages: Error: 500 status code (no body) InternalServerError: 500 status code (no body) at APIError.generate (file:///app/node_modules/openai/error.mjs:62:20) at OpenAI.makeStatusError (file:///app/node_modules/openai/core.mjs:268:25) at OpenAI.makeRequest (file:///app/node_modules/openai/core.mjs:311:30) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async GET (file:///app/build/server/chunks/_server.ts-Bh-eBlsA.js:11:16) at async render_endpoint (file:///app/build/server/index.js:1228:20) at async resolve2 (file:///app/build/server/index.js:3708:22) at async csp (file:///app/build/server/chunks/hooks.server-BtaMPuC-.js:92:20) at async respond (file:///app/build/server/index.js:3603:22) at async fetch (file:///app/build/server/index.js:3360:26) { status: 500, headers: { 'content-length': '46', 'content-type': 'application/json', date: 'Tue, 27 Aug 2024 19:32:46 GMT', server: 'envoy', 'x-envoy-upstream-service-time': '18' }, request_id: undefined, error: undefined, code: undefined, param: undefined, type: undefined } Error fetching assistants InternalServerError: 500 status code (no body) at APIError.generate (file:///app/node_modules/openai/error.mjs:62:20) at OpenAI.makeStatusError (file:///app/node_modules/openai/core.mjs:268:25) at OpenAI.makeRequest (file:///app/node_modules/openai/core.mjs:311:30) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async GET (file:///app/build/server/chunks/_server.ts-DJXll3aH.js:12:28) at async render_endpoint (file:///app/build/server/index.js:1228:20) at async resolve2 (file:///app/build/server/index.js:3708:22) at async csp (file:///app/build/server/chunks/hooks.server-BtaMPuC-.js:92:20) at async respond (file:///app/build/server/index.js:3603:22) at async fetch (file:///app/build/server/index.js:3360:26) { status: 500, headers: { 'content-length': '46', 'content-type': 'application/json', date: 'Tue, 27 Aug 2024 19:32:46 GMT', server: 'envoy', 'x-envoy-upstream-service-time': '18' }, request_id: undefined, error: undefined, code: undefined, param: undefined, type: undefined }



### Additional Context

Add any other context or screenshots about the technical debt here.

defenseunicorns / leapfrogai