search

QuivrHQ / quivr

Opiniated RAG for integrating GenAI in your apps 🧠 Focus on your product rather than the RAG. Easy integration in existing products with customisation! Any LLM: GPT4, Groq, Llama. Any Vectorstore: PGVector, Faiss. Any Files. Anyway you want.
https://core.quivr.com
Other
36.54k stars 3.56k forks source link

[Bug]: supabase_vector_secondbrain container is not ready: unhealthy #3206

Open windowshopr opened 1 month ago

windowshopr commented 1 month ago

What happened?

Windows 10, Python 3.11, Docker Desktop.

Trying to follow the documentation step by step.

cd backend && supabase start
supabase_vector_secondbrain container is not ready: unhealthy
Try rerunning the command with --debug to troubleshoot the error.
supabase start --debug
Pruned containers: [c0808a31c4edefd350ab4453a12a4c9c311f3be4a350cb898e90b461c0490a07 875ad8a524b0a4e8e8ceaef129008da8302d5bf2c67c2382264992e8c94cb655]
Pruned volumes: [supabase_db_secondbrain supabase_config_secondbrain]
Pruned network: [supabase_network_secondbrain]
failed to create docker container: Error response from daemon: Conflict. The container name "/supabase_vector_secondbrain" is already in use by container "a1ff8452be7d314a0eed6d054981708a1d66181d5e904b34e710715aa0946888". You have to remove (or rename) that container to be able to reuse that name.

image

I noticed that the supabase vector container is stuck on restarting, so I delete that container, and re-reun:

supabase start --debug

Same output as above:

supabase_vector_secondbrain container is not ready: unhealthy

Something about vector_secondbrain not passing the health checks?

image

Relevant log output

Looking at the Docker Desktop log of the supabase_vector_secondbrain output, there's this, not sure if it helps debug?

...
2024-09-16 00:40:20 2024-09-16T06:40:20.054112Z ERROR source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Listing currently running containers failed. error=error trying to connect: tcp connect error: Network unreachable (os error 101)
2024-09-16 00:40:20 2024-09-16T06:40:20.056385Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:9001 playground=http://0.0.0.0:9001/playground
2024-09-16 00:40:20 2024-09-16T06:40:20.056545Z  INFO vector::app: All sources have finished.
2024-09-16 00:40:20 2024-09-16T06:40:20.056645Z  INFO vector: Vector has stopped.
2024-09-16 00:40:20 2024-09-16T06:40:20.069990Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="rest_logs, logflare_storage, logflare_db, project_logs, logflare_realtime, kong_logs, logflare_kong, auth_logs, router, db_logs, docker_host, realtime_logs, storage_logs, logflare_rest, logflare_functions, kong_err, logflare_auth" time_remaining="59 seconds left"
2024-09-16 00:40:26 2024-09-16T06:40:26.845553Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-09-16 00:40:26 2024-09-16T06:40:26.845651Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2024-09-16 00:40:26 2024-09-16T06:40:26.845706Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2024-09-16 00:40:26 2024-09-16T06:40:26.872608Z  WARN vector::config::loading: Transform "router._unmatched" has no consumers
2024-09-16 00:40:26 2024-09-16T06:40:26.873113Z  INFO source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Capturing logs from now on. now=2024-09-16T06:40:26.873062835+00:00
2024-09-16 00:40:26 2024-09-16T06:40:26.873193Z  INFO source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Listening to docker log events.
2024-09-16 00:40:27 2024-09-16T06:40:27.030058Z  INFO vector::topology::running: Running healthchecks.
2024-09-16 00:40:27 2024-09-16T06:40:27.030427Z  INFO vector: Vector has started. debug="false" version="0.28.1" arch="x86_64" revision="ff15924 2023-03-06"
2024-09-16 00:40:27 2024-09-16T06:40:27.030556Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:27 2024-09-16T06:40:27.030681Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:27 2024-09-16T06:40:27.030749Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:27 2024-09-16T06:40:27.030818Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:27 2024-09-16T06:40:27.030926Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:27 2024-09-16T06:40:27.031079Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:27 2024-09-16T06:40:27.031146Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:27 2024-09-16T06:40:27.035574Z ERROR source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Listing currently running containers failed. error=error trying to connect: tcp connect error: Network unreachable (os error 101)
2024-09-16 00:40:27 2024-09-16T06:40:27.039825Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:9001 playground=http://0.0.0.0:9001/playground
2024-09-16 00:40:27 2024-09-16T06:40:27.039863Z  INFO vector::app: All sources have finished.
2024-09-16 00:40:27 2024-09-16T06:40:27.039866Z  INFO vector: Vector has stopped.
2024-09-16 00:40:27 2024-09-16T06:40:27.050768Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="logflare_kong, logflare_rest" time_remaining="59 seconds left"
2024-09-16 00:40:40 2024-09-16T06:40:40.186820Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-09-16 00:40:40 2024-09-16T06:40:40.186894Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2024-09-16 00:40:40 2024-09-16T06:40:40.187112Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2024-09-16 00:40:40 2024-09-16T06:40:40.214866Z  WARN vector::config::loading: Transform "router._unmatched" has no consumers
2024-09-16 00:40:40 2024-09-16T06:40:40.215238Z  INFO source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Capturing logs from now on. now=2024-09-16T06:40:40.215191780+00:00
2024-09-16 00:40:40 2024-09-16T06:40:40.215288Z  INFO source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Listening to docker log events.
2024-09-16 00:40:40 2024-09-16T06:40:40.382769Z  INFO vector::topology::running: Running healthchecks.
2024-09-16 00:40:40 2024-09-16T06:40:40.382984Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:40 2024-09-16T06:40:40.383056Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:40 2024-09-16T06:40:40.383063Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:40 2024-09-16T06:40:40.383065Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:40 2024-09-16T06:40:40.383067Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:40 2024-09-16T06:40:40.383068Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:40 2024-09-16T06:40:40.383070Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-16 00:40:40 2024-09-16T06:40:40.383033Z  INFO vector: Vector has started. debug="false" version="0.28.1" arch="x86_64" revision="ff15924 2023-03-06"
2024-09-16 00:40:40 2024-09-16T06:40:40.388487Z ERROR source{component_kind="source" component_id=docker_host component_type=docker_logs component_name=docker_host}: vector::sources::docker_logs: Listing currently running containers failed. error=error trying to connect: tcp connect error: Network unreachable (os error 101)
2024-09-16 00:40:40 2024-09-16T06:40:40.393704Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:9001 playground=http://0.0.0.0:9001/playground
2024-09-16 00:40:40 2024-09-16T06:40:40.393806Z  INFO vector::app: All sources have finished.
2024-09-16 00:40:40 2024-09-16T06:40:40.393831Z  INFO vector: Vector has stopped.
2024-09-16 00:40:40 2024-09-16T06:40:40.405276Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="logflare_kong, logflare_realtime, storage_logs, logflare_storage" time_remaining="59 seconds left"


### Twitter / LinkedIn details

_No response_
linear[bot] commented 1 month ago

CORE-202 [Bug]: supabase_vector_secondbrain container is not ready: unhealthy

dosubot[bot] commented 1 month ago

Hey there, @windowshopr! 👋 I'm here to help you with any bugs, questions, or contributions you might have. Let's squash those bugs and get things running smoothly!

I found a similar issue that might be relevant to your problem: [Bug]: dependency failed to start: container supabase-analytics is unhealthy. This issue is closed and was last updated on August 09, 2024 [1].

For the supabase_vector_secondbrain container, ensure that the necessary environment variables are correctly set. These include:

Additionally, the backend/ci-migration.sh script indicates the need for:

Make sure these variables are correctly configured in your environment.

If the issue persists, you might want to check the Docker logs for more specific error messages related to the health checks. The logs you provided indicate a network issue (Network unreachable (os error 101)). This could be related to Docker's network configuration or a firewall setting on your Windows machine.

Try the following steps:

  1. Ensure Docker Desktop is up to date.
  2. Restart Docker Desktop.
  3. Check your firewall settings to ensure Docker has the necessary permissions.
  4. Verify that no other containers are conflicting with the supabase_vector_secondbrain container.

If these steps do not resolve the issue, you may need to investigate further into the Docker network settings or consult the Docker documentation for more detailed troubleshooting steps.

To continue talking to Dosu, mention @dosu.

windowshopr commented 1 month ago

The similar issue you found didn't provide an answer.

The .env file is left completely default, except for the uncommenting of the line:

OLLAMA_API_BASE_URL=http://host.docker.internal:11434 # Uncomment to activate ollama. This is the local url for the ollama api

...as I plan to use Ollama models for my applications.

The documentation should be updated if other changes are to be made for the supabase instructions.

As a second attempt watching the installation video on Youtube doesn't show using supabase at all, just:

docker compose pull
docker compose up

...Which results in the following output, where I see two questionable sections

time="2024-09-16T00:57:53-06:00" level=warning msg="I:\\nasty\\Python_Projects\\LLM\\quivr\\docker-compose.yml: `version` is obsolete"
[+] Running 7/0
 ✔ Container redis        Created                                                                                  0.0s
 ✔ Container backend-api  Created                                                                                  0.0s
 ✔ Container web          Created                                                                                  0.0s
 ✔ Container beat         Created                                                                                  0.0s
 ✔ Container worker       Created                                                                                  0.0s
 ✔ Container flower       Created                                                                                  0.0s
 ✔ Container notifier     Created                                                                                  0.0s
Attaching to backend-api, beat, flower, notifier, redis, web, worker
redis        | 1:C 16 Sep 2024 06:57:54.279 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis        | 1:C 16 Sep 2024 06:57:54.279 * Redis version=7.2.3, bits=64, commit=00000000, modified=0, pid=1, just started
redis        | 1:C 16 Sep 2024 06:57:54.279 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis        | 1:M 16 Sep 2024 06:57:54.280 * monotonic clock: POSIX clock_gettime
redis        | 1:M 16 Sep 2024 06:57:54.284 * Running mode=standalone, port=6379.
redis        | 1:M 16 Sep 2024 06:57:54.289 * Server initialized
redis        | 1:M 16 Sep 2024 06:57:54.289 * Loading RDB produced by version 7.2.3
redis        | 1:M 16 Sep 2024 06:57:54.289 * RDB age 47 seconds
redis        | 1:M 16 Sep 2024 06:57:54.289 * RDB memory usage when created 1.44 Mb
redis        | 1:M 16 Sep 2024 06:57:54.290 * Done loading RDB, keys loaded: 12, keys expired: 0.
redis        | 1:M 16 Sep 2024 06:57:54.290 * DB loaded from disk: 0.001 seconds
redis        | 1:M 16 Sep 2024 06:57:54.290 * Ready to accept connections tcp
backend-api  | INFO:     Will watch for changes in these directories: ['/app']
backend-api  | INFO:     Uvicorn running on http://0.0.0.0:5050 (Press CTRL+C to quit)
backend-api  | INFO:     Started reloader process [1] using StatReload
web          |    ▲ Next.js 14.1.0
web          |    - Local:        http://localhost:3000
web          |    - Network:      http://0.0.0.0:3000
web          |
web          |  ✓ Ready in 379ms
notifier     | [INFO] quivr_api.modules.dependencies [dependencies.py:179]: Creating Supabase client
notifier     | [INFO] notifier_service [celery_monitor.py:150]: Started  quivr-notifier service...
notifier     | [INFO] notifier_service [celery_monitor.py:51]: Initialized knowledge_service. Listening to task event...
flower       | [INFO] quivr_api.modules.dependencies [dependencies.py:179]: Creating Supabase client
beat         | [INFO] quivr_api.modules.dependencies [dependencies.py:179]: Creating Supabase client
worker       | [INFO] quivr_api.modules.dependencies [dependencies.py:179]: Creating Supabase client
flower       | [INFO] quivr_api.modules.dependencies [dependencies.py:153]: Creating Postgres DB engine
flower       | [I 240916 06:58:03 command:168] Visit me at http://0.0.0.0:5555
flower       | [I 240916 06:58:03 command:176] Broker: redis://redis:6379/0
flower       | [I 240916 06:58:03 command:177] Registered tasks:
flower       |     ['NotionConnectorLoad',
flower       |      'celery.accumulate',
flower       |      'celery.backend_cleanup',
flower       |      'celery.chain',
flower       |      'celery.chord',
flower       |      'celery.chord_unlock',
flower       |      'celery.chunks',
flower       |      'celery.group',
flower       |      'celery.map',
flower       |      'celery.starmap',
flower       |      'check_is_premium_task',
flower       |      'fetch_and_store_notion_files_task',
worker       | [INFO] quivr_api.modules.dependencies [dependencies.py:153]: Creating Postgres DB engine
flower       |      'process_active_syncs_task',
beat         | [INFO] quivr_api.modules.dependencies [dependencies.py:153]: Creating Postgres DB engine
flower       |      'process_crawl_task',
flower       |      'process_file_task',
flower       |      'process_notion_sync_task',
flower       |      'process_sync_task',
worker       | /usr/local/lib/python3.11/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
flower       |      'quivr_worker.celery_worker.ping_telemetry',
worker       | absolutely not recommended!
beat         | [2024-09-16 06:58:03,672: INFO/MainProcess] beat: Starting...
flower       |      'quivr_worker.celery_worker.process_integration_brain_sync_user_brain']
worker       |
flower       | [I 240916 06:58:03 mixins:229] Connected to redis://redis:6379/0
worker       | Please specify a different user using the --uid option.
worker       |
worker       | User information: uid=0 euid=0 gid=0 egid=0
worker       |
worker       |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
worker       |
worker       |  -------------- celery@f5355532b0d8 v5.4.0 (opalescent)
worker       | --- ***** -----
worker       | -- ******* ---- Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.31 2024-09-16 06:58:03
worker       | - *** --- * ---
beat         | [2024-09-16 06:58:03,774: INFO/MainProcess] Scheduler: Sending due task process_premium_users (check_is_premium_task)
worker       | - ** ---------- [config]
worker       | - ** ---------- .> app:         quivr_api.celery_config:0x7fc4242c4790
beat         | [2024-09-16 06:58:03,793: INFO/MainProcess] Scheduler: Sending due task process_active_syncs (process_active_syncs_task)
worker       | - ** ---------- .> transport:   redis://redis:6379/0
worker       | - ** ---------- .> results:     redis://redis:6379/0
worker       | - *** --- * --- .> concurrency: 2 (prefork)
worker       | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
worker       | --- ***** -----
worker       |  -------------- [queues]
worker       |                 .> celery           exchange=celery(direct) key=celery
worker       |
worker       |
worker       | [tasks]
worker       |   . NotionConnectorLoad
worker       |   . check_is_premium_task
worker       |   . fetch_and_store_notion_files_task
worker       |   . process_active_syncs_task
worker       |   . process_crawl_task
worker       |   . process_file_task
worker       |   . process_notion_sync_task
worker       |   . process_sync_task
worker       |   . quivr_worker.celery_worker.ping_telemetry
worker       |   . quivr_worker.celery_worker.process_integration_brain_sync_user_brain
worker       |
worker       | [2024-09-16 06:58:04,032: WARNING/MainProcess] /usr/local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
worker       | whether broker connection retries are made during startup in Celery 6.0 and above.
worker       | If you wish to retain the existing behavior for retrying connections on startup,
worker       | you should set broker_connection_retry_on_startup to True.
worker       |   warnings.warn(
worker       |
worker       | [2024-09-16 06:58:04,061: INFO/MainProcess] Connected to redis://redis:6379/0
worker       | [2024-09-16 06:58:04,064: WARNING/MainProcess] /usr/local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
worker       | whether broker connection retries are made during startup in Celery 6.0 and above.
worker       | If you wish to retain the existing behavior for retrying connections on startup,
worker       | you should set broker_connection_retry_on_startup to True.
worker       |   warnings.warn(
worker       |
worker       | [2024-09-16 06:58:04,070: INFO/MainProcess] mingle: searching for neighbors
backend-api  | [INFO] quivr_api.modules.dependencies [dependencies.py:179]: Creating Supabase client
backend-api  | [INFO] quivr_api.modules.dependencies [dependencies.py:153]: Creating Postgres DB engine
flower       | [W 240916 06:58:04 inspector:44] Inspect method stats failed
flower       | [W 240916 06:58:04 inspector:44] Inspect method registered failed
flower       | [W 240916 06:58:04 inspector:44] Inspect method active_queues failed
flower       | [W 240916 06:58:04 inspector:44] Inspect method active failed
flower       | [W 240916 06:58:04 inspector:44] Inspect method reserved failed
flower       | [W 240916 06:58:04 inspector:44] Inspect method scheduled failed
worker       | [2024-09-16 06:58:05,078: INFO/MainProcess] mingle: all alone
worker       | [2024-09-16 06:58:05,091: INFO/MainProcess] celery@f5355532b0d8 ready.
worker       | [2024-09-16 06:58:05,095: INFO/MainProcess] Task check_is_premium_task[7f027681-655a-4596-9a42-c411da568b29] received
worker       | [2024-09-16 06:58:05,097: INFO/MainProcess] Task process_active_syncs_task[f857b54f-b2ec-49d3-ba84-7a34dc0cd7a9] received
worker       | [INFO] celery_worker [check_premium.py:16]: Stripe deactivated, skipping check for premium users
worker       | [2024-09-16 06:58:05,103: INFO/ForkPoolWorker-2] Task check_is_premium_task[7f027681-655a-4596-9a42-c411da568b29] succeeded in 0.004031250000025466s: None
worker       | [2024-09-16 06:58:05,146: ERROR/ForkPoolWorker-1] Task process_active_syncs_task[f857b54f-b2ec-49d3-ba84-7a34dc0cd7a9] raised unexpected: ConnectionRefusedError(111, "Connect call failed ('192.168.65.254', 54322)")
worker       | Traceback (most recent call last):
worker       |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task
worker       |     R = retval = fun(*args, **kwargs)
worker       |                  ^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__
worker       |     return self.run(*args, **kwargs)
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/app/worker/quivr_worker/celery_worker.py", line 270, in process_active_syncs_task
worker       |     loop.run_until_complete(
worker       |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
worker       |     return future.result()
worker       |            ^^^^^^^^^^^^^^^
worker       |   File "/app/worker/quivr_worker/syncs/process_active_syncs.py", line 62, in process_all_active_syncs
worker       |     async with build_syncs_utils(sync_services) as mapping_sync_utils:
worker       |   File "/usr/local/lib/python3.11/contextlib.py", line 204, in __aenter__
worker       |     return await anext(self.gen)
worker       |            ^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/app/worker/quivr_worker/syncs/utils.py", line 90, in build_syncs_utils
worker       |     raise e
worker       |   File "/app/worker/quivr_worker/syncs/utils.py", line 56, in build_syncs_utils
worker       |     await session.execute(
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlmodel/ext/asyncio/session.py", line 143, in execute
worker       |     return await super().execute(
worker       |            ^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/session.py", line 461, in execute
worker       |     result = await greenlet_spawn(
worker       |              ^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 201, in greenlet_spawn
worker       |     result = context.throw(*sys.exc_info())
worker       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlmodel/orm/session.py", line 127, in execute
worker       |     return super().execute(
worker       |            ^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
worker       |     return self._execute_internal(
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
worker       |     conn = self._connection_for_bind(bind)
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
worker       |     return trans._connection_for_bind(engine, execution_options)
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "<string>", line 2, in _connection_for_bind
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
worker       |     ret_value = fn(self, *arg, **kw)
worker       |                 ^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
worker       |     conn = bind.connect()
worker       |            ^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
worker       |     return self._connection_cls(self)
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
worker       |     self._dbapi_connection = engine.raw_connection()
worker       |                              ^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
worker       |     return self.pool.connect()
worker       |            ^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 449, in connect
worker       |     return _ConnectionFairy._checkout(self)
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
worker       |     fairy = _ConnectionRecord.checkout(pool)
worker       |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
worker       |     rec = pool._do_get()
worker       |           ^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
worker       |     with util.safe_reraise():
flower       | [W 240916 06:58:05 inspector:44] Inspect method revoked failed
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
flower       | [W 240916 06:58:05 inspector:44] Inspect method conf failed
worker       |     raise exc_value.with_traceback(exc_tb)
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
worker       |     return self._create_connection()
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
worker       |     return _ConnectionRecord(self)
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
worker       |     self.__connect()
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
worker       |     with util.safe_reraise():
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
worker       |     raise exc_value.with_traceback(exc_tb)
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
worker       |     self.dbapi_connection = connection = pool._invoke_creator(self)
worker       |                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 643, in connect
worker       |     return dialect.connect(*cargs, **cparams)
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 621, in connect
worker       |     return self.loaded_dbapi.connect(*cargs, **cparams)
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 932, in connect
worker       |     await_only(creator_fn(*arg, **kw)),
worker       |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
worker       |     return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
worker       |     value = await result
worker       |             ^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connection.py", line 2329, in connect
worker       |     return await connect_utils._connect(
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 1017, in _connect
worker       |     raise last_error or exceptions.TargetServerAttributeNotMatched(
worker       |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 991, in _connect
worker       |     conn = await _connect_addr(
worker       |            ^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 828, in _connect_addr
worker       |     return await __connect_addr(params, True, *args)
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 873, in __connect_addr
worker       |     tr, pr = await connector
worker       |              ^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 744, in _create_ssl_connection
worker       |     tr, pr = await loop.create_connection(
worker       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
worker       |     raise exceptions[0]
worker       |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
worker       |     sock = await self._connect_sock(
worker       |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
worker       |     await self.sock_connect(sock, address)
worker       |   File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
worker       |     return await fut
worker       |            ^^^^^^^^^
worker       |   File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
worker       |     raise OSError(err, f'Connect call failed {address}')
worker       | ConnectionRefusedError: [Errno 111] Connect call failed ('192.168.65.254', 54322)
backend-api  | [INFO] quivr_api.main [main.py:120]: Telemetry enabled, we use telemetry to collect anonymous usage data.
backend-api  | [INFO] quivr_api.main [main.py:121]: To disable telemetry, set the TELEMETRY_ENABLED environment variable to false.
worker       | [2024-09-16 06:58:08,540: INFO/MainProcess] Events of group {task} enabled by remote.
backend-api  | INFO:     Started server process [8]
backend-api  | INFO:     Waiting for application startup.
backend-api  | INFO:     Application startup complete.
backend-api  | INFO:     127.0.0.1:57108 - "GET /healthz HTTP/1.1" 500 Internal Server Error
backend-api  | ERROR:    Exception in ASGI application
backend-api  | Traceback (most recent call last):
backend-api  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
backend-api  |     result = await app(  # type: ignore[func-returns-value]
backend-api  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
backend-api  |     return await self.app(scope, receive, send)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
backend-api  |     await super().__call__(scope, receive, send)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
backend-api  |     await self.middleware_stack(scope, receive, send)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
backend-api  |     raise exc
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
backend-api  |     await self.app(scope, receive, _send)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
backend-api  |     await self.app(scope, receive, send)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
backend-api  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
backend-api  |     raise exc
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
backend-api  |     await app(scope, receive, sender)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 754, in __call__
backend-api  |     await self.middleware_stack(scope, receive, send)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 774, in app
backend-api  |     await route.handle(scope, receive, send)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 295, in handle
backend-api  |     await self.app(scope, receive, send)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
backend-api  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
backend-api  |     raise exc
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
backend-api  |     await app(scope, receive, sender)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
backend-api  |     response = await f(request)
backend-api  |                ^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 269, in app
backend-api  |     solved_result = await solve_dependencies(
backend-api  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/fastapi/dependencies/utils.py", line 596, in solve_dependencies
backend-api  |     solved = await solve_generator(
backend-api  |              ^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/fastapi/dependencies/utils.py", line 521, in solve_generator
backend-api  |     return await stack.enter_async_context(cm)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/contextlib.py", line 638, in enter_async_context
backend-api  |     result = await _enter(cm)
backend-api  |              ^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/contextlib.py", line 204, in __aenter__
backend-api  |     return await anext(self.gen)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/app/api/quivr_api/modules/dependencies.py", line 108, in get_async_session
backend-api  |     raise e
backend-api  |   File "/app/api/quivr_api/modules/dependencies.py", line 101, in get_async_session
backend-api  |     await session.execute(
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlmodel/ext/asyncio/session.py", line 143, in execute
backend-api  |     return await super().execute(
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/ext/asyncio/session.py", line 461, in execute
backend-api  |     result = await greenlet_spawn(
backend-api  |              ^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 201, in greenlet_spawn
backend-api  |     result = context.throw(*sys.exc_info())
backend-api  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlmodel/orm/session.py", line 127, in execute
backend-api  |     return super().execute(
backend-api  |            ^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2362, in execute
backend-api  |     return self._execute_internal(
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2237, in _execute_internal
backend-api  |     conn = self._connection_for_bind(bind)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2106, in _connection_for_bind
backend-api  |     return trans._connection_for_bind(engine, execution_options)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "<string>", line 2, in _connection_for_bind
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
backend-api  |     ret_value = fn(self, *arg, **kw)
backend-api  |                 ^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1189, in _connection_for_bind
backend-api  |     conn = bind.connect()
backend-api  |            ^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3278, in connect
backend-api  |     return self._connection_cls(self)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
backend-api  |     self._dbapi_connection = engine.raw_connection()
backend-api  |                              ^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3302, in raw_connection
backend-api  |     return self.pool.connect()
backend-api  |            ^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 449, in connect
backend-api  |     return _ConnectionFairy._checkout(self)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
backend-api  |     fairy = _ConnectionRecord.checkout(pool)
backend-api  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
backend-api  |     rec = pool._do_get()
backend-api  |           ^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
backend-api  |     with util.safe_reraise():
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
backend-api  |     raise exc_value.with_traceback(exc_tb)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
backend-api  |     return self._create_connection()
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
backend-api  |     return _ConnectionRecord(self)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
backend-api  |     self.__connect()
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
backend-api  |     with util.safe_reraise():
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
backend-api  |     raise exc_value.with_traceback(exc_tb)
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
backend-api  |     self.dbapi_connection = connection = pool._invoke_creator(self)
backend-api  |                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 643, in connect
backend-api  |     return dialect.connect(*cargs, **cparams)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 621, in connect
backend-api  |     return self.loaded_dbapi.connect(*cargs, **cparams)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 932, in connect
backend-api  |     await_only(creator_fn(*arg, **kw)),
backend-api  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
backend-api  |     return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
backend-api  |     value = await result
backend-api  |             ^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connection.py", line 2329, in connect
backend-api  |     return await connect_utils._connect(
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 1017, in _connect
backend-api  |     raise last_error or exceptions.TargetServerAttributeNotMatched(
backend-api  |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 991, in _connect
backend-api  |     conn = await _connect_addr(
backend-api  |            ^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 828, in _connect_addr
backend-api  |     return await __connect_addr(params, True, *args)
backend-api  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 873, in __connect_addr
backend-api  |     tr, pr = await connector
backend-api  |              ^^^^^^^^^^^^^^^
backend-api  |   File "/usr/local/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 744, in _create_ssl_connection
backend-api  |     tr, pr = await loop.create_connection(
backend-api  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-api  |   File "uvloop/loop.pyx", line 2039, in create_connection
backend-api  |   File "uvloop/loop.pyx", line 2016, in uvloop.loop.Loop.create_connection
backend-api  | ConnectionRefusedError: [Errno 111] Connection refused

Trying to ignore these issues, I navigate to the webpage anyway and try to login:

image

No dice, connection refused.