`supabase start` Error: service not healthy

[probablykasper]

Bug report

• [x] I confirm this is a bug with Supabase, not with my own application.
• [x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

supabase start fails after updating to the latest version.

To Reproduce

supabase start

Logs

supabase start --debug logs:

[...]
supabase_pg_meta_limey container logs:

> @supabase/postgres-meta@0.0.0-automated start
> node dist/server/app.js

(node:36) ExperimentalWarning: Importing JSON modules is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
2023/05/09 23:57:31 Sent Header: Host [docker]
2023/05/09 23:57:31 Sent Header: User-Agent [Go-http-client/1.1]
2023/05/09 23:57:31 Send Done
2023/05/09 23:57:31 Recv First Byte
supabase_studio_limey container logs:
info  - Loaded env from /app/studio/.env
Listening on port 3000
Error: service not healthy: [realtime-dev.supabase_realtime_limey supabase_pg_meta_limey supabase_studio_limey]

System information

• OS: macOS 13.3.1
• RAM allocated to Docker: 12 GB

[cohlar]

Same for me on Windows 11 after I upgraded from version 1.55.1 to 1.60.2.

[jramiroz98]

Just found the same bug

[jramiroz98]

Guys! This worked for me pnpx supabase start --ignore-health-check

[romain-sen]

I have the issue :

$ npx supabase start        
supabase_rest_my-app container logs:
12/Jun/2023:14:09:05 +0000: Attempting to connect to the database...
12/Jun/2023:14:09:05 +0000: Connection successful
12/Jun/2023:14:09:05 +0000: Listening on port 3000
12/Jun/2023:14:09:05 +0000: Config reloaded
12/Jun/2023:14:09:05 +0000: Listening for notifications on the pgrst channel
12/Jun/2023:14:09:05 +0000: Schema cache loaded
12/Jun/2023:14:09:05 +0000: Attempting to connect to the database...
12/Jun/2023:14:09:05 +0000: Connection successful
12/Jun/2023:14:09:05 +0000: Config reloaded
12/Jun/2023:14:09:05 +0000: Schema cache loaded
12/Jun/2023:14:09:07 +0000: Attempting to connect to the database...
12/Jun/2023:14:09:07 +0000: Connection successful
12/Jun/2023:14:09:07 +0000: Config reloaded
12/Jun/2023:14:09:07 +0000: Schema cache loaded
12/Jun/2023:14:09:07 +0000: Attempting to connect to the database...
12/Jun/2023:14:09:07 +0000: Connection successful
12/Jun/2023:14:09:07 +0000: Config reloaded
12/Jun/2023:14:09:07 +0000: Schema cache loaded
12/Jun/2023:14:09:07 +0000: Attempting to connect to the database...
12/Jun/2023:14:09:07 +0000: Connection successful
12/Jun/2023:14:09:07 +0000: Config reloaded
12/Jun/2023:14:09:07 +0000: Schema cache loaded
supabase_deno_relay_my-app container logs:
Download https://deno.land/std@0.182.0/http/server.ts
Download https://deno.land/x/jose@v4.13.1/index.ts
Download https://deno.land/x/jose@v4.13.1/jwe/compact/decrypt.ts
[...]
Download https://deno.land/x/jose@v4.13.1/util/base64url.ts
Download https://deno.land/std@0.182.0/async/mod.ts
Download https://deno.land/x/jose@v4.13.1/lib/buffer_utils.ts
[...]
Download https://deno.land/x/jose@v4.13.1/runtime/generate.ts
Download https://deno.land/std@0.182.0/async/abortable.ts
[...]
Download https://deno.land/std@0.182.0/async/retry.ts
Download https://deno.land/x/jose@v4.13.1/lib/check_iv_length.ts
Download https://deno.land/x/jose@v4.13.1/runtime/check_cek_length.ts
Download https://deno.land/x/jose@v4.13.1/runtime/timing_safe_equal.ts
[...]
Download https://deno.land/x/jose@v4.13.1/lib/check_p2s.ts
Serving functions on http://localhost:54321/functions/v1/<function-name>
Error: service not healthy: [supabase_rest_my-app supabase_deno_relay_my-app]
Try rerunning the command with --debug to troubleshoot the error.

Debug bring no useful information and the flag --ignore-health-check allows me to finish supabase start, but then I can't query my db from my nextjs app without getting the error : An invalid response was received from the upstream server.

I am sure that I have enough ram (32gb). I tried to restart every container or delete and reinstall them but nothing work. Impossible to use the supabase cli... Anyone to help me ?

[romain-sen]

I destroy all my docker container & images, and now it works.

[evelant]

Possibly related to https://github.com/supabase/cli/issues/1177

[romain-sen]

but even if it works with --ignore-health-check flag, the api doesn't work. I can run my db tests and it works. The studio works too, but the api does not. This does not help much.

[munkacsimark]

I have similar issue, every container is starting correctly except supabase_studio_xxxxx. The container keeps restarting then I'm getting the service not healthy error. --ignore-health-check flag works but this way I cannot start the studio locally. This log entry is repeating in the container:

info  - Loaded env from /app/studio/.env
Listening on port 3000
No storage option exists to persist the session, which may result in unexpected behavior when using auth.
        If you want to set persistSession to true, please provide a storage option or you may set persistSession to false to disable this warning.

Any suggestion for this issue?

[NiklasPor]

As a workaround downgrading to 1.69.0 worked for me. There's probably a more up-to-date version which also works. With the latest version I'm experiencing exactly the same issue as @munkacsimark on my mac.

[IbtisamBeik]

I have same issue as @munkacsimark on my mac

[bllchmbrs]

I have the same issue on my mac or at least one that manifests like this issue or this issue: https://github.com/supabase/cli/issues/778

I am on supabase@1.82.6

Here's what I tried, so others try not to go down the same path:

1. Uninstalled brew version and installed NPM version
2. Removed supabase dir and re-'init'.
3. Tried turning off other services like edge runtime that I don't need.
4. Tried installing to 1.69, didn't work for me.
5. Upping CPUs for docker desktop install 4 => 6, RAM 8 => 12.
6. All other services launch fine, it's just the studio, which of course, is the thing I want to use and test.
7. Tried --ignore-health-check, launches everything else but the studio.
8. Tried nuking all the containers / images and that doesn't work.
9. Tried restarting my machine.

[NiklasPor]

As a workaround downgrading to 1.69.0 worked for me. There's probably a more up-to-date version which also works. With the latest version I'm experiencing exactly the same issue as @munkacsimark on my mac.

Actually I still have this error, even on the latest version 1.88.0

[willdavidow]

As a workaround downgrading to 1.69.0 worked for me. There's probably a more up-to-date version which also works. With the latest version I'm experiencing exactly the same issue as @munkacsimark on my mac.

Actually I still have this error, even on the latest version 1.88.0

This should not be closed. Also still seeing this error using the latest cli version 1.88.0 (and the previous version when this issue was closed). Using --ignore-health-check still lets me around the issue so I can at least work locally.

[thomergil]

I did a binary search between 1.69.0 and 1.88.0 and found that 1.72.0 works, and 1.73.0 no longer does. So 1.72.0 is the most recent version that does not exhibit this problem. Something broke between 1.72.0 and 1.73.0.

yarn remove supabase
yarn add supabase@1.72.0 --save-dev
yarn run supabase stop
yarn run supabase start

[sweatybridge]

Thanks! This is really helpful. There studio version changed in 1.73.0 from 20230605 to 20230622 so there should be some commits between these dates that caused the crash loop on studio.

[sweatybridge]

I think the major change between these versions is that studio is bumped from node16 to node18. It also seems like a platform specific issue.

@thomergil could you tell me more about your OS, arch, and docker version to help us reproduce?

[NiklasPor]

FYI I'm experiencing this on:

• Intel MacOS (Ventura)
• Docker v4.21.1 (5 CPUs, 16 GB RAM)
• Package manager: pnpm v8.6.11

[thomergil]

Problem only happens on my Intel Mac. Not on M1 Mac laptop... You may be right about platform. 🙂

[bllchmbrs]

I'm on intel i5, Ventura, as well.

On Wed, Aug 23, 2023 at 3:02 AM Thomer Gil @.***> wrote:

Only happens on my Intel Mac. Not on laptop... 🙂

— Reply to this email directly, view it on GitHub https://github.com/supabase/cli/issues/1083#issuecomment-1689668807, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMRAB4DLRNPUTVR2Z25UU3XWXIMTANCNFSM6AAAAAAX34JZNA . You are receiving this because you commented.Message ID: @.***>

[willdavidow]

If it helps, these are my system specs:

[sweatybridge]

I'm using an Intel based Mac without errors.. Could you also check the docker settings?

[sweatybridge]

UPDATE: I managed to reproduce this error after upgrading docker desktop to 4.21.1 (docker engine 24.0.1). I was previously on 4.12.0 (engine 20.10.17).

I will dig through their changelog to see if there's any fixes.

UPDATE 2: The last known working version is 4.18.0

[sweatybridge]

Found the upstream issue for this segfault https://github.com/docker/for-mac/issues/6824. If you are on Intel Mac with docker desktop versions >= 4.19.0, the workaround is to disable MacOS Virtualization Framework.

Closing this issue as unrelated to CLI.

[thomergil]

Thank you @sweatybridge!

[munkacsimark]

Thanks a lot guys!

[bllchmbrs]

Thank you for this!

[MortensenC]

@sweatybridge, you are a genius! Many thanks

[myhendry]

Found the upstream issue for this segfault docker/for-mac#6824. If you are on Intel Mac with docker desktop versions >= 4.19.0, the workaround is to disable MacOS Virtualization Framework.

Closing this issue as unrelated to CLI.

Hi. I updated my docker desktop to version 4.25.2 and now it doesnt allow the user to uncheck the Use Virtualization framework checkbox. On my docker desktop, there is a footnote " Uses Virtualization framework for creating and managing Docker Desktop Linux VM in macOS 12.5 and above." As I'm now unable to disable Use Virtualization framework, im getting this message "service not healthy: [realtime-dev.supabase_realtime_server supabase_storage_server supabase_pg_meta_server supabase_studio_server]" when i run npx supabase start on my computer. Previously on an earlier version of docker desktop, i was able to uncheck Use Virtualization framework and everything was working nicely. Issue resurfaced when I upgraded my docker desktop

Anyone has any solution to resolve this issue "service not healthy: [realtime-dev.supabase_realtime_server supabase_storage_server supabase_pg_meta_server supabase_studio_server]" now that I can't disable Uses Virtualization framework on docker desktop? Thanks

[philsherry]

Do we know if there’s a way to do this in code? Rather than have to physically check a box on my Intel Macs, it would be nicer if I could just add something like this:

if [[ ${CPU} == "x86_64" ]]; then
  MacOS_Virtualization_Framework=false
fi

[Onat-T]

I don't know why this is a closed issue. I have to uninstall Docker now in February 2024, Mac M1 chip. Docker goes in a loop. Maybe this is a Docker issue, but I never had to change my settings and cause Docker to crash. This only happened because of "service not healthy" issues.

That being said, thank you for having supabase open source! Thank you for all your hard work. I appreciate how tough it is.

Just for FYI though, this is what the suggestions in this thread put my Docker into:

[kiptoomm]

I am still facing this issue. My environment:

• macOS 11.7
• docker v24.0.6 (this is the latest version I can run since I can't upgrade to any newer version. See https://github.com/docker/for-mac/issues/4996#issuecomment-1836731668 and https://github.com/docker/for-mac/issues/6824 for the background)
• supabase CLI v1.148.6

The error persists even after trying the popular solution about disabling MacOS Virtualization Framework, and also this: https://github.com/docker/for-mac/issues/6824#issuecomment-1726939831 (changing VirtioFS to osxfs) but neither seemed to work.

Detailed console log error

``` $ supabase start supabase_auth_my_project container logs: {"level":"info","msg":"Go runtime metrics collection started","time":"2024-03-12T08:32:19Z"} {"args":["20240115144230_remove_ip_address_from_saml_relay_state.sql"],"component":"pop","level":"warning","msg":"ignoring file %s because it does not match the migration file pattern","time":"2024-03-12T08:32:19Z"} {"args":[0.138821722],"component":"pop","level":"info","msg":"%.4f seconds","time":"2024-03-12T08:32:19Z"} {"level":"fatal","msg":"running db migrations: error executing /usr/local/etc/auth/migrations/20221208132122_backfill_email_last_sign_in_at.up.sql, sql: -- previous backfill migration left last_sign_in_at to be null, which broke some projects\n\ndo $$\nbegin\nupdate auth.identities\n set last_sign_in_at = '2022-11-25'\n where\n last_sign_in_at is null and\n created_at = '2022-11-25' and\n updated_at = '2022-11-25' and\n provider = 'email' and\n id = user_id::text;\nend $$;\n: ERROR: operator does not exist: uuid = text (SQLSTATE 42883)","time":"2024-03-12T08:32:19Z"} {"level":"info","msg":"Go runtime metrics collection started","time":"2024-03-12T08:32:21Z"} {"args":["20240115144230_remove_ip_address_from_saml_relay_state.sql"],"component":"pop","level":"warning","msg":"ignoring file %s because it does not match the migration file pattern","time":"2024-03-12T08:32:21Z"} {"args":[0.138929058],"component":"pop","level":"info","msg":"%.4f seconds","time":"2024-03-12T08:32:21Z"} {"level":"fatal","msg":"running db migrations: error executing /usr/local/etc/auth/migrations/20221208132122_backfill_email_last_sign_in_at.up.sql, sql: -- previous backfill migration left last_sign_in_at to be null, which broke some projects\n\ndo $$\nbegin\nupdate auth.identities\n set last_sign_in_at = '2022-11-25'\n where\n last_sign_in_at is null and\n created_at = '2022-11-25' and\n updated_at = '2022-11-25' and\n provider = 'email' and\n id = user_id::text;\nend $$;\n: ERROR: operator does not exist: uuid = text (SQLSTATE 42883)","time":"2024-03-12T08:32:21Z"} {"level":"info","msg":"Go runtime metrics collection started","time":"2024-03-12T08:32:23Z"} {"args":["20240115144230_remove_ip_address_from_saml_relay_state.sql"],"component":"pop","level":"warning","msg":"ignoring file %s because it does not match the migration file pattern","time":"2024-03-12T08:32:23Z"} {"args":[0.163714446],"component":"pop","level":"info","msg":"%.4f seconds","time":"2024-03-12T08:32:23Z"} {"level":"fatal","msg":"running db migrations: error executing /usr/local/etc/auth/migrations/20221208132122_backfill_email_last_sign_in_at.up.sql, sql: -- previous backfill migration left last_sign_in_at to be null, which broke some projects\n\ndo $$\nbegin\nupdate auth.identities\n set last_sign_in_at = '2022-11-25'\n where\n last_sign_in_at is null and\n created_at = '2022-11-25' and\n updated_at = '2022-11-25' and\n provider = 'email' and\n id = user_id::text;\nend $$;\n: ERROR: operator does not exist: uuid = text (SQLSTATE 42883)","time":"2024-03-12T08:32:23Z"} {"level":"info","msg":"Go runtime metrics collection started","time":"2024-03-12T08:32:26Z"} {"args":["20240115144230_remove_ip_address_from_saml_relay_state.sql"],"component":"pop","level":"warning","msg":"ignoring file %s because it does not match the migration file pattern","time":"2024-03-12T08:32:26Z"} {"args":[0.119528471],"component":"pop","level":"info","msg":"%.4f seconds","time":"2024-03-12T08:32:26Z"} {"level":"fatal","msg":"running db migrations: error executing /usr/local/etc/auth/migrations/20221208132122_backfill_email_last_sign_in_at.up.sql, sql: -- previous backfill migration left last_sign_in_at to be null, which broke some projects\n\ndo $$\nbegin\nupdate auth.identities\n set last_sign_in_at = '2022-11-25'\n where\n last_sign_in_at is null and\n created_at = '2022-11-25' and\n updated_at = '2022-11-25' and\n provider = 'email' and\n id = user_id::text;\nend $$;\n: ERROR: operator does not exist: uuid = text (SQLSTATE 42883)","time":"2024-03-12T08:32:26Z"} {"level":"info","msg":"Go runtime metrics collection started","time":"2024-03-12T08:32:28Z"} {"args":["20240115144230_remove_ip_address_from_saml_relay_state.sql"],"component":"pop","level":"warning","msg":"ignoring file %s because it does not match the migration file pattern","time":"2024-03-12T08:32:28Z"} {"args":[0.213032821],"component":"pop","level":"info","msg":"%.4f seconds","time":"2024-03-12T08:32:29Z"} {"level":"fatal","msg":"running db migrations: error executing /usr/local/etc/auth/migrations/20221208132122_backfill_email_last_sign_in_at.up.sql, sql: -- previous backfill migration left last_sign_in_at to be null, which broke some projects\n\ndo $$\nbegin\nupdate auth.identities\n set last_sign_in_at = '2022-11-25'\n where\n last_sign_in_at is null and\n created_at = '2022-11-25' and\n updated_at = '2022-11-25' and\n provider = 'email' and\n id = user_id::text;\nend $$;\n: ERROR: operator does not exist: uuid = text (SQLSTATE 42883)","time":"2024-03-12T08:32:29Z"} {"level":"info","msg":"Go runtime metrics collection started","time":"2024-03-12T08:32:33Z"} {"args":["20240115144230_remove_ip_address_from_saml_relay_state.sql"],"component":"pop","level":"warning","msg":"ignoring file %s because it does not match the migration file pattern","time":"2024-03-12T08:32:33Z"} {"args":[0.245803205],"component":"pop","level":"info","msg":"%.4f seconds","time":"2024-03-12T08:32:33Z"} {"level":"fatal","msg":"running db migrations: error executing /usr/local/etc/auth/migrations/20221208132122_backfill_email_last_sign_in_at.up.sql, sql: -- previous backfill migration left last_sign_in_at to be null, which broke some projects\n\ndo $$\nbegin\nupdate auth.identities\n set last_sign_in_at = '2022-11-25'\n where\n last_sign_in_at is null and\n created_at = '2022-11-25' and\n updated_at = '2022-11-25' and\n provider = 'email' and\n id = user_id::text;\nend $$;\n: ERROR: operator does not exist: uuid = text (SQLSTATE 42883)","time":"2024-03-12T08:32:33Z"} {"level":"info","msg":"Go runtime metrics collection started","time":"2024-03-12T08:32:40Z"} {"args":["20240115144230_remove_ip_address_from_saml_relay_state.sql"],"component":"pop","level":"warning","msg":"ignoring file %s because it does not match the migration file pattern","time":"2024-03-12T08:32:40Z"} {"args":[0.161462524],"component":"pop","level":"info","msg":"%.4f seconds","time":"2024-03-12T08:32:40Z"} {"level":"fatal","msg":"running db migrations: error executing /usr/local/etc/auth/migrations/20221208132122_backfill_email_last_sign_in_at.up.sql, sql: -- previous backfill migration left last_sign_in_at to be null, which broke some projects\n\ndo $$\nbegin\nupdate auth.identities\n set last_sign_in_at = '2022-11-25'\n where\n last_sign_in_at is null and\n created_at = '2022-11-25' and\n updated_at = '2022-11-25' and\n provider = 'email' and\n id = user_id::text;\nend $$;\n: ERROR: operator does not exist: uuid = text (SQLSTATE 42883)","time":"2024-03-12T08:32:40Z"} {"level":"info","msg":"Go runtime metrics collection started","time":"2024-03-12T08:32:48Z"} {"args":["20240115144230_remove_ip_address_from_saml_relay_state.sql"],"component":"pop","level":"warning","msg":"ignoring file %s because it does not match the migration file pattern","time":"2024-03-12T08:32:48Z"} {"args":[0.116117724],"component":"pop","level":"info","msg":"%.4f seconds","time":"2024-03-12T08:32:48Z"} {"level":"fatal","msg":"running db migrations: error executing /usr/local/etc/auth/migrations/20221208132122_backfill_email_last_sign_in_at.up.sql, sql: -- previous backfill migration left last_sign_in_at to be null, which broke some projects\n\ndo $$\nbegin\nupdate auth.identities\n set last_sign_in_at = '2022-11-25'\n where\n last_sign_in_at is null and\n created_at = '2022-11-25' and\n updated_at = '2022-11-25' and\n provider = 'email' and\n id = user_id::text;\nend $$;\n: ERROR: operator does not exist: uuid = text (SQLSTATE 42883)","time":"2024-03-12T08:32:48Z"} service not healthy: [supabase_auth_my_project] Try rerunning the command with --debug to troubleshoot the error. ```

Ultimately, I have to manually delete all my containers/images/volumes and re-run supabase start to set up a fresh local dev environment. But I know this is not a sustainable solution as the issue keeps recurring randomly. @sweatybridge any guidance?

[sweatybridge]

Ultimately, I have to manually delete all my containers/images/volumes and re-run supabase start to set up a fresh local dev environment.

Have you tried supabase stop --no-backup? It should delete the volumes so it's easier to start from fresh.

[kiptoomm]

Ultimately, I have to manually delete all my containers/images/volumes and re-run supabase start to set up a fresh local dev environment.

Have you tried supabase stop --no-backup? It should delete the volumes so it's easier to start from fresh.

I now have, and that indeed seems to be an easier way to cleanly reset the database. However, I'm still curious whether there's any longer-term solution or plans to fix it on the supabase CLI side for those of us where disabling the macOS virtualization as a workaround hasn't made any difference...

[ahmadawais]

This fixed my problem, thank you @kiptoomm

npx supabase stop --no-backup
npx supabase start

[Nurex-ui]

I just had this, been working on idx, worked first run but then after closing and reopening the repo 2 or so days later, running supabase start got me this. What worked for me is removing images and reinitializing everything fresh.