cannot set up supabase self host. Almost every deployment does not start

[lb-ovc]

Bug report

Nothing works.

Describe the bug

I am trying to setup selfhost since around 8 hours now. Tried everything and nothing works.

This is how it currently looks:

A clear and concise description of what the bug is.

To Reproduce

Clone and edit the values.yaml following exactly the "Customize" section of the documentation.

Config i did:

SMTP_ADMIN_MAIL = "blabla@blabla.de"
SMTP_HOST="strato"
SMTP_PORT="463"
SMTP_SENDER_NAME="blabla"

Generated the jwt secrets like told

secret:
  jwt:
    anonKey: eyJhbGci...3ZO2g
    serviceKey:eyJhbGciOi...NA10c5A
    secret: 4RgG4u...Q==

set the smpt secret like told:

secret:
  smtp:
    username: blabla@blabla.de
    password: secretpasswordcensored

set the db secret:

secret:
  db:
    username: supabase
    password: supersecretpasswordcensored
    database: supabase
   host: 10.....0 # added this line to point to my external database running on a hetzner server (installing this on master which can access the local ip address of the db server) 
   port: 5432 # added this also
   # specify existing secret, which takes precedence over variables above
   secretRef: ""
    # override secret keys for existing secret refs
   secretRefKey:
     username: username
     password: password
     database: database
     host: host
     port: port

disabled the db image because i have the external database

db:
   enabled: false

Updated the dashboard secret

secrets:
  dashboard:
    username: myusername
    password: supersecurecensoredpassword
    # specify existing secret, which takes precedence over variables above
    # secretRef: ""
    # override secret keys for existing secret refs
    secretRefKey:
      username: username
      password: password

generated a key with the openssl rand 64 | base64 command for the analytics key

 analytics:
    apiKey: "8Zzj+...UhiQ=="
    # specify existing secret, which takes precedence over variable above
    # override secret keys for existing secret refs
    secretRefKey:
      apiKey: apiKey

we are using exoscale for the s3 so i did this config:

secrets:
  s3:
    keyId: "xyz"
    accessKey: "LW...zq0"
    # specify existing secret, which takes precedence over variables above
    # secretRef: ""
    # override secret keys for existing secret refs
    secretRefKey:
      keyId: keyId
      accessKey: accessKey

and

storage: 
environment:
    # Override the database hostname if using external database
    DB_HOST: 10.xxxxx.5
    DB_USER: supabase_storage_admin # does this need to be changed to my supabase user from my external db??
    DB_PORT: 5432
    DB_DRIVER: postgres
    DB_SSL: disable  # disable, allow, prefer, require, verify-ca, verify-full
    PGOPTIONS: -c search_path=storage,public
    FILE_SIZE_LIMIT: "52428800"
    STORAGE_BACKEND: file  # file, s3
    FILE_STORAGE_BACKEND_PATH: /var/lib/storage
    TENANT_ID: stub
    REGION: stub
    GLOBAL_S3_BUCKET: app
    # Set variables below and secret.s3 above to enable S3 storage bucket
    # If using this chart's minio, skip the endpoint and protocol below
    STORAGE_BACKEND: s3
    GLOBAL_S3_ENDPOINT: https://sos-de-fra-1.exo.io
    GLOBAL_S3_PROTOCOL: https
    GLOBAL_S3_FORCE_PATH_STYLE: true

the last thing was setting every example.com in the value.yaml to https://supabase.mywebsite.de and for the kong config the host to supabase.mywebsite.de

Copied the config to my server and ran helm install bfh -f config.yaml .

Output:

root@master-1:~/supabase-kubernetes/charts/supabase# helm install bfh -f config.yaml .
walk.go:74: found symbolic link in path: /root/supabase-kubernetes/charts/supabase/ci/example-values.yaml resolves to /root/supabase-kubernetes/charts/supabase/values.example.yaml. Contents of linked file included and used
W0728 21:27:57.968324  795993 warnings.go:70] spec.template.spec.containers[0].env[35].name: duplicate name "GOTRUE_SMTP_PASS"
NAME: bfh
LAST DEPLOYED: Sun Jul 28 21:27:52 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
---
Thank you for installing supabase!

Visit the Studio dashboard at http://supabase.mywebsite.de

initDB returns "accepting Connections"

Expected behavior

Services and pods to start without an error.

A clear and concise description of what you expected to happen.

Screenshots

So here is the log from every failed deployments:

analytics: bfh-supabase-analytics-6c8ffcd7cc-2s42t_supabase-analytics.log

auth: Cant even download the logs

functions:

meta:

realtime: bfh-supabase-realtime-5dd674d7d5-whlbh_supabase-realtime.log

rest: bfh-supabase-rest-6fdf6ccd44-wmbwn_supabase-rest.log

storage: bfh-supabase-storage-6574cf9844-7zwpv_supabase-storage.log

vector:

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

This is just really frustrating having a so bad documented documentation for such a cool product like supabase...

[lb-ovc]

I have also tried to use the example.values.yaml which works better with 3 services not starting (realtime, auth, analytics) Analytics and auth same issue as above realtime: bfh-supabase-realtime-58d4495f5c-b8nvg_supabase-realtime.log

[jweberde]

@lb-ovc can you also share how you setup the hetzner database server. The server requires certain extension installed and if I recall correctly you need to install them manually.

Compare https://github.com/supabase-community/supabase-kubernetes/blob/main/charts/supabase/docker/Database.Dockerfile

[lb-ovc]

These are the servers. They are all in a network so they can comunicate between eachother.

They have Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-105-generic x86_64) installed.

the db server has postgres 14.12 installed

Ill try to install the extensions in a second and update you as i did not do that

[lb-ovc]

@jweberde

whats the best way for me to install this manually? I would guess it needs to be done on my db server, right?

[jweberde]

@jweberde

whats the best way for me to install this manually? I would guess it needs to be done on my db server, right?

From the Dockerfile I would run it on your Database Hosts as root user. Also check the github projects for their requirements.

https://github.com/michelp/pgjwt https://github.com/eulerto/wal2json

For example maybe it is possible to use the wal2json apt package =>

$ sudo apt-get install postgresql-16-wal2json

I also extracted from a local project (via supabase cli) the extension list of my currently running project.

-- run as postgres user.
select * from  pg_available_extensions where installed_version is not null;

name	default_version	installed_version	comment
pgcrypto	1.3	1.3	cryptographic functions
plpgsql	1.0	1.0	PL/pgSQL procedural language
unaccent	1.1	1.1	text search dictionary that removes accents
moddatetime	1.0	1.0	functions for tracking last modification time
uuid-ossp	1.1	1.1	generate universally unique identifiers (UUIDs)
pg_stat_statements	1.10	1.10	track planning and execution statistics of all SQL statements executed
pg_graphql	1.5.4	1.5.4	pg_graphql: GraphQL support
pg_net	0.8.0	0.8.0	Async HTTP
supabase_vault	0.2.8	0.2.8	Supabase Vault Extension
pgjwt	0.2.0	0.2.0	JSON Web Token API for Postgresql
pgsodium	3.1.8	3.1.8	Postgres extension for libsodium functions

NOTE: wal2json is a system extension not a postgres extension. Thats why it is not visible I guess.

[lb-ovc]

Also another question. Do i need to assing all example.com to the same url? is Kong doing the rest?

[jweberde]

If you are using the kubernetes version I would "recommend" doing that way otherwise you need to create Ingress with the same configuration and map it:

2 Things to look out for as I am going through the same process right now:

1.Kong YAML has a known bug: https://github.com/supabase-community/supabase-kubernetes/pull/75 so really clone the repository and fix it before using it.

config.yaml in the chart/templates/kong directory.

2. I personally do not like to cohost the studio on the same url as the website endpoint, because I prefer additional security for the studio. For example different network more secure security than basic authentication. My goal is to disable the catchall kong route using an Ingress whitelisting (serve only /pg /analytics /functions ...) and then move the dashboard to my extra secured mgmt ingress endpoint (maybe even onto my local kubernetes cluster), which has built in login via forward auth and google workplace.

I haven't done that yet but it is on my todo list. what is important in general if you want to use the studio on remote at least activate basic authentication and some kind of bruteforce prevention.

[lb-ovc]

So i have installed in now like this on my db server:

sudo apt-get install make git gcc libc6-dev
git clone https://github.com/michelp/pgjwt.git
cd pgjwt && make install

git clone https://github.com/eulerto/wal2json.git
cd wal2json
make
sudo make install

then

sudo systemctl restart postgresql

[lb-ovc]

pgjwt

pgjwt is in the extensions folder located at /usr/shared/postgresql/14/extensions but wal2json is not

[jweberde]

Then give it a try and push your migrations, to create a runnable setup:

Push new migrations to the remote database

Usage:
  supabase db push [flags]

Flags:
      --db-url string     Pushes to the database specified by the connection string (must be percent-encoded).
      --dry-run           Print the migrations that would be applied, but don't actually apply them.
  -h, --help              help for push
      --include-all       Include all migrations not found on remote history table.
      --include-roles     Include custom roles from supabase/roles.sql.
      --include-seed      Include seed data from supabase/seed.sql.
      --linked            Pushes to the linked project. (default true)
      --local             Pushes to the local database.
  -p, --password string   Password to your remote Postgres database.

DB URL EXAMPLE --db-url postgresql://postgres:postgres@localhost:54321/postgres

Full discloser I tried it that way using the built in db and still had some issues of a missing database. I than researched in the cli:

https://github.com/supabase/cli/blob/develop/pkg/migration/history.go#L27

SET LOCAL lock_timeout = '4s';
CREATE SCHEMA IF NOT EXISTS supabase_migrations;
CREATE TABLE IF NOT EXISTS supabase_migrations.schema_migrations (version text NOT NULL PRIMARY KEY);
ALTER TABLE supabase_migrations.schema_migrations ADD COLUMN IF NOT EXISTS statements text[];
ALTER TABLE supabase_migrations.schema_migrations ADD COLUMN IF NOT EXISTS name text;

And ran the above commands manually. The idea is that the first time you push a migration the migration table should be created. I got an error that he could not do the diff because of missing migration tables.

[lb-ovc]

Where do i execute the supabase command? tried it on the master but command "supabase" is not found

[jweberde]

From your local project on your local computer for example using a tunnel.

https://supabase.com/docs/reference/cli/supabase-init

For example for kubernetes running locally on your PC in this case using MAC Terminal:

kubectl port-forward -n your-namespace $(kubectl get pods -n your-namespace -o name -l app.kubernetes.io/name=supabase-db) 5432:5432

SSH

ssh -i /path/to/your/private_key -L 5432:localhost:5432 your_username@remote_host -N

Then the db url would be localhost:5432 and you can you use

npx supabase db push

within your project root.

If you want we can chat on https://discord.com/ just add me jweber_de and we can figure it out and than document the solution here.

[lb-ovc]

From your local project on your local computer for example using a tunnel.

https://supabase.com/docs/reference/cli/supabase-init

For example for kubernetes running locally on your PC in this case using MAC Terminal:

kubectl port-forward -n your-namespace $(kubectl get pods -n your-namespace -o name -l app.kubernetes.io/name=supabase-db) 5432:5432

SSH

ssh -i /path/to/your/private_key -L 5432:localhost:5432 your_username@remote_host -N

Then the db url would be localhost:5432 and you can you use

npx supabase db push

within your project root.

If you want we can chat on https://discord.com/ just add me jweber_de and we can figure it out and than document the solution here.

I have send you a friend request :)

[AhmadMasri]

hi there, thanks for your comments it helps me a lot with the initial setup however I'm still not able to run all the deployments, I wonder if you could find a solution ?

[lb-ovc]

hi there, thanks for your comments it helps me a lot with the initial setup however I'm still not able to run all the deployments, I wonder if you could find a solution ?

Hey mate, yeah i couldnt get it to work with kubernetes due to the bad documentation this repo has.

I just switched to running supabase with docker swarm as kubernetes was a bit overkill anyways for our purpose.

Ill hope if we need to convert to kubernetes in the future that it will be better documented here.

[AhmadMasri]

hi there, thanks for your comments it helps me a lot with the initial setup however I'm still not able to run all the deployments, I wonder if you could find a solution ?

Hey mate, yeah i couldnt get it to work with kubernetes due to the bad documentation this repo has.

I just switched to running supabase with docker swarm as kubernetes was a bit overkill anyways for our purpose.

Ill hope if we need to convert to kubernetes in the future that it will be better documented here.

i just had a successful run on K8s with the Bitnami chart after struggling with the configuration. However, some components (edge function, vector, imgproxy, and analytics) were missing.

[rowan-vr]

I also ran into lots of issues whilst trying to set it up, most due to ambiguous configuration keys in the values.yaml, after lots of trail and error I got most pods running aside database connections, which still results in an completely useless deployment :(