Closed perroboc closed 6 months ago
Could you share the output from docker ps -a
after manually starting with --ignore-health-check
? For eg.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2cbd18c4fa18 public.ecr.aws/supabase/studio:20230803-15c6762 "docker-entrypoint.s…" 28 seconds ago Up 27 seconds (healthy) 0.0.0.0:54323->3000/tcp supabase_studio_cli
5389387c0c50 public.ecr.aws/supabase/postgres-meta:v0.68.0 "docker-entrypoint.s…" 28 seconds ago Up 28 seconds (healthy) 8080/tcp supabase_pg_meta_cli
760d64064256 public.ecr.aws/supabase/edge-runtime:v1.11.1 "sh -c 'mkdir -p /ho…" 29 seconds ago Up 28 seconds 8081/tcp supabase_edge_runtime_cli
387333881b54 public.ecr.aws/supabase/imgproxy:v3.8.0 "imgproxy" 30 seconds ago Up 29 seconds (healthy) 8080/tcp storage_imgproxy_cli
fc4800fb93af public.ecr.aws/supabase/storage-api:v0.40.4 "docker-entrypoint.s…" 31 seconds ago Up 30 seconds (healthy) 5000/tcp supabase_storage_cli
7cefb6c7b7bd public.ecr.aws/supabase/postgrest:v11.1.0 "/bin/postgrest" 31 seconds ago Up 31 seconds 3000/tcp supabase_rest_cli
119af862feb7 public.ecr.aws/supabase/realtime:v2.10.1 "/usr/bin/tini -s -g…" 32 seconds ago Up 31 seconds (healthy) 4000/tcp realtime-dev.supabase_realtime_cli
f56b6dc8d94c public.ecr.aws/supabase/gotrue:v2.92.1 "gotrue" 32 seconds ago Up 32 seconds (healthy) 9999/tcp supabase_auth_cli
588a14c54533 public.ecr.aws/supabase/kong:2.8.1 "sh -c 'cat <<'EOF' …" 33 seconds ago Up 32 seconds (healthy) 8001/tcp, 8443-8444/tcp, 0.0.0.0:54321->8000/tcp supabase_kong_cli
3448b09c885c public.ecr.aws/supabase/postgres:15.1.0.103 "sh -c 'cat <<'EOF' …" 46 seconds ago Up 45 seconds (healthy) 0.0.0.0:54322->5432/tcp supabase_db_cli
If all services are reported as healthy, then it just means that services take a longer time to start up on your host system.
Hi there, sorry for resurrecting this old thread. I encountered the same problem as OP. If you're still willing to give a look, here's the output from podman ps -a
:
> podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f10fef95111 public.ecr.aws/supabase/postgres:15.1.0.117 postgres -c confi... About a minute ago Up About a minute (healthy) 0.0.0.0:54322->5432/tcp supabase_db_buoj
7fd9cde979a0 public.ecr.aws/supabase/kong:2.8.1 52 seconds ago Up 52 seconds (healthy) 0.0.0.0:54321->8000/tcp supabase_kong_buoj
7794dcd29671 public.ecr.aws/supabase/gotrue:v2.92.1 gotrue 52 seconds ago Up 52 seconds (healthy) supabase_auth_buoj
da66f184eb9f public.ecr.aws/supabase/inbucket:3.0.3 -logjson 51 seconds ago Up 52 seconds (healthy) 0.0.0.0:54324->9000/tcp supabase_inbucket_buoj
f23a9f73f0f5 public.ecr.aws/supabase/realtime:v2.10.1 /bin/sh -c /app/b... 51 seconds ago Up 52 seconds (unhealthy) realtime-dev.supabase_realtime_buoj
d81cfc0b72ca public.ecr.aws/supabase/postgrest:v11.2.0 /bin/postgrest 51 seconds ago Up 52 seconds supabase_rest_buoj
bac50714d41d public.ecr.aws/supabase/storage-api:v0.40.4 node dist/server.... 51 seconds ago Up 51 seconds (healthy) supabase_storage_buoj
6835128fc9c6 public.ecr.aws/supabase/imgproxy:v3.8.0 imgproxy 51 seconds ago Up 51 seconds (healthy) storage_imgproxy_buoj
e89a1e421536 public.ecr.aws/supabase/postgres-meta:v0.68.0 npm run start 50 seconds ago Up 51 seconds (unhealthy) supabase_pg_meta_buoj
131dcc9c973d public.ecr.aws/supabase/studio:20230921-d657f29 node studio/serve... 50 seconds ago Up 51 seconds (unhealthy) 0.0.0.0:54323->3000/tcp supabase_studio_buoj
Here are the error messages I get with healthchecks enabled:
23:33:02.003 [error] Postgrex.Protocol (#PID<0.137.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (supabase_db_buoj:5432): non-existing domain - :nxdomain
23:33:02.003 [error] Postgrex.Protocol (#PID<0.136.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (supabase_db_buoj:5432): non-existing domain - :nxdomain
23:33:08.976 [debug] QUERY ERROR queue=14988.0ms
begin []
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 14984ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
1. Ensuring your database is available and that you can connect to it
2. Tracking down slow queries and making sure they are running fast enough
3. Increasing the pool_size (although this increases resource consumption)
4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
See DBConnection.start_link/2 for more information
(db_connection 2.4.2) lib/db_connection.ex:904: DBConnection.transaction/3
/app/lib/realtime-2.10.1/priv/repo/seeds.exs:5: (file)
nofile:1: (file)
(stdlib 4.3) erl_eval.erl:748: :erl_eval.do_apply/7
(elixir 1.14.3) lib/code.ex:425: Code.validated_eval_string/3
23:33:17.456 [error] Postgrex.Protocol (#PID<0.137.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (supabase_db_buoj:5432): non-existing domain - :nxdomain
23:33:17.456 [error] Postgrex.Protocol (#PID<0.136.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (supabase_db_buoj:5432): non-existing domain - :nxdomain
23:33:24.439 [error] Could not create schema migrations table. This error usually happens due to the following:
* The database does not exist
* The "schema_migrations" table, which Ecto uses for managing
migrations, was defined by another library
* There is a deadlock while migrating (such as using concurrent
indexes with a migration_lock)
To fix the first issue, run "mix ecto.create".
To address the second, you can run "mix ecto.drop" followed by
"mix ecto.create". Alternatively you may configure Ecto to use
another table and/or repository for managing migrations:
config :realtime, Realtime.Repo,
migration_source: "some_other_table_for_schema_migrations",
migration_repo: AnotherRepoForSchemaMigrations
The full error report is shown below.
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 14990ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
1. Ensuring your database is available and that you can connect to it
2. Tracking down slow queries and making sure they are running fast enough
3. Increasing the pool_size (although this increases resource consumption)
4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
See DBConnection.start_link/2 for more information
(ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1
(elixir 1.14.3) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
(ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:1024: Ecto.Adapters.SQL.execute_ddl/4
(ecto_sql 3.8.3) lib/ecto/migrator.ex:696: Ecto.Migrator.verbose_schema_migration/3
(ecto_sql 3.8.3) lib/ecto/migrator.ex:510: Ecto.Migrator.lock_for_migrations/4
(ecto_sql 3.8.3) lib/ecto/migrator.ex:422: Ecto.Migrator.run/4
(ecto_sql 3.8.3) lib/ecto/migrator.ex:146: Ecto.Migrator.with_repo/3
nofile:1: (file)
2023/10/15 19:33:29 Sent Header: Host [api.moby.localhost]
2023/10/15 19:33:29 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2023/10/15 19:33:29 Send Done
2023/10/15 19:33:29 Recv First Byte
supabase_rest_buoj container logs:
15/Oct/2023:23:32:52 +0000: Starting PostgREST 11.2.0...
15/Oct/2023:23:32:52 +0000: Attempting to connect to the database...
15/Oct/2023:23:32:52 +0000: Connection successful
15/Oct/2023:23:32:52 +0000: Listening on port 3000
15/Oct/2023:23:32:52 +0000: Config reloaded
15/Oct/2023:23:32:52 +0000: Listening for notifications on the pgrst channel
15/Oct/2023:23:32:52 +0000: Schema cache loaded
15/Oct/2023:23:32:52 +0000: Starting PostgREST 11.2.0...
15/Oct/2023:23:32:52 +0000: Attempting to connect to the database...
15/Oct/2023:23:32:58 +0000: {"code":"PGRST000","details":"could not translate host name \"supabase_db_buoj\" to address: Try again\n","hint":null,"message":"Database connection error. Retrying the connection."}
15/Oct/2023:23:32:58 +0000: Attempting to reconnect to the database in 0 seconds...
10.89.0.6 - anon [15/Oct/2023:23:32:58 +0000] "HEAD / HTTP/1.1" 503 - "" "Go-http-client/1.1"
10.89.0.6 - anon [15/Oct/2023:23:33:03 +0000] "HEAD / HTTP/1.1" 503 - "" "Go-http-client/1.1"
15/Oct/2023:23:33:04 +0000: {"code":"PGRST000","details":"could not translate host name \"supabase_db_buoj\" to address: Try again\n","hint":null,"message":"Database connection error. Retrying the connection."}
15/Oct/2023:23:33:04 +0000: Attempting to reconnect to the database in 1 seconds...
10.89.0.6 - anon [15/Oct/2023:23:33:08 +0000] "HEAD / HTTP/1.1" 503 - "" "Go-http-client/1.1"
15/Oct/2023:23:33:11 +0000: {"code":"PGRST000","details":"could not translate host name \"supabase_db_buoj\" to address: Try again\n","hint":null,"message":"Database connection error. Retrying the connection."}
15/Oct/2023:23:33:11 +0000: Attempting to reconnect to the database in 2 seconds...
10.89.0.6 - anon [15/Oct/2023:23:33:14 +0000] "HEAD / HTTP/1.1" 503 - "" "Go-http-client/1.1"
10.89.0.6 - anon [15/Oct/2023:23:33:19 +0000] "HEAD / HTTP/1.1" 503 - "" "Go-http-client/1.1"
15/Oct/2023:23:33:20 +0000: {"code":"PGRST000","details":"could not translate host name \"supabase_db_buoj\" to address: Try again\n","hint":null,"message":"Database connection error. Retrying the connection."}
15/Oct/2023:23:33:20 +0000: Attempting to reconnect to the database in 4 seconds...
10.89.0.6 - anon [15/Oct/2023:23:33:24 +0000] "HEAD / HTTP/1.1" 503 - "" "Go-http-client/1.1"
10.89.0.6 - anon [15/Oct/2023:23:33:29 +0000] "HEAD / HTTP/1.1" 503 - "" "Go-http-client/1.1"
2023/10/15 19:33:29 Sent Header: Host [api.moby.localhost]
2023/10/15 19:33:29 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2023/10/15 19:33:29 Send Done
2023/10/15 19:33:29 Recv First Byte
supabase_pg_meta_buoj container logs:
> @supabase/postgres-meta@0.0.0-automated start
> node dist/server/server.js
(node:31) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
{"level":"info","time":"2023-10-15T23:32:54.998Z","pid":31,"hostname":"e2d63c401563","msg":"Server listening at http://0.0.0.0:8080"}
{"level":"info","time":"2023-10-15T23:32:55.004Z","pid":31,"hostname":"e2d63c401563","msg":"Server listening at http://0.0.0.0:8081"}
2023/10/15 19:33:29 Sent Header: Host [api.moby.localhost]
2023/10/15 19:33:29 Sent Header: User-Agent [Docker-Client/unknown-version (linux)]
2023/10/15 19:33:29 Send Done
2023/10/15 19:33:29 Recv First Byte
supabase_studio_buoj container logs:
info - Loaded env from /app/studio/.env
Listening on port 3000
This browser does not support the Navigator Locks API. Please update it.
I can confirm that I'm facing the exact same issue on fedora:
$ sudo supabase start --ignore-health-check
[...]
service not healthy: [realtime-dev.supabase_realtime_backend supabase_pg_meta_backend supabase_studio_backend]
Started supabase local development setup.
API URL: http://127.0.0.1:54321
GraphQL URL: http://127.0.0.1:54321/graphql/v1
DB URL: postgresql://postgres:postgres@127.0.0.1:54322/postgres
Studio URL: http://127.0.0.1:54323
Inbucket URL: http://127.0.0.1:54324
JWT secret: super-secret-jwt-token-with-at-least-32-characters-long
[...]
$ sudo podman ps
81ac6f864eff public.ecr.aws/supabase/postgres:15.1.0.131 postgres -c confi... 4 minutes ago Up 4 minutes (healthy) 0.0.0.0:54322->5432/tcp supabase_db_backend
d6545594dd7f public.ecr.aws/supabase/kong:2.8.1 4 minutes ago Up 4 minutes (healthy) 0.0.0.0:54321->8000/tcp supabase_kong_backend
7891311ca1d0 public.ecr.aws/supabase/gotrue:v2.104.2 gotrue 4 minutes ago Up 4 minutes (healthy) supabase_auth_backend
85320b94cbd8 public.ecr.aws/supabase/inbucket:3.0.3 -logjson 4 minutes ago Up 4 minutes (healthy) 0.0.0.0:54324->9000/tcp supabase_inbucket_backend
71cdb220cb50 public.ecr.aws/supabase/realtime:v2.10.1 /bin/sh -c /app/b... 4 minutes ago Up 4 minutes (unhealthy) realtime-dev.supabase_realtime_backend
1307cfb601e8 public.ecr.aws/supabase/postgrest:v11.2.0 /bin/postgrest 4 minutes ago Up 4 minutes supabase_rest_backend
de3ce0061392 public.ecr.aws/supabase/storage-api:v0.43.9 node dist/server.... 4 minutes ago Up 4 minutes (healthy) supabase_storage_backend
75f50f07dd96 public.ecr.aws/supabase/imgproxy:v3.8.0 imgproxy 4 minutes ago Up 4 minutes (healthy) storage_imgproxy_backend
cb305e1915ea public.ecr.aws/supabase/edge-runtime:v1.22.3 4 minutes ago Up 4 minutes supabase_edge_runtime_backend
c310de303b40 public.ecr.aws/supabase/postgres-meta:v0.68.0 npm run start 4 minutes ago Up 4 minutes (unhealthy) supabase_pg_meta_backend
91e917d58567 public.ecr.aws/supabase/studio:20231023-7e2cd92 node studio/serve... 4 minutes ago Up 4 minutes (unhealthy) 0.0.0.0:54323->3000/tcp supabase_studio_backend
When checking logs for these 3 pods there is no error to be found. The Studio is working but most pages are empty so the backend is probably malfunctioning.
This seems to be an issue with upstream podman as other applications also faced similar issues https://elixirforum.com/t/postgrex-error-nxdomain-when-resolving-rootless-docker-compose-database-domain/50662.
Perhaps there's a podman network config that needs to be updated or use docker as a workaround.
closing as duplicate of https://github.com/supabase/cli/issues/265
Describe the bug Using podman with supabase, I get these errors on launch: service not healthy: [realtime-dev.supabase_realtime_X supabase_pg_meta_X supabase_studio_X]. A workaround is to start supabase ignoring the health check.
To Reproduce Steps to reproduce the behavior:
Workaround:
supabase start --ignore-health-check
Expected behavior I expected for the healthchecks to be OK
Screenshots Not applicable
Desktop (please complete the following information):
Additional context
/etc/hosts
~/.config/containers/containers.conf