[Bug]: Realtime in Supabase is not working - Coolify overrides a required container_name in docker-compose.yml

[actraiser]

Description

When using realtime in Supabase, the Kong-Container receives the request from the client and tries to connect to realtime-dev.supabase-realtime but cannot resolve this. I can see in the docker-compose-file that there is the correct "container_name" under the realtime-section:

  realtime-dev:
    image: 'supabase/realtime:v2.28.23'
    container_name: realtime-dev.supabase-realtime

But when switching to the "show deployable docker compose" view, this is tranformed to:

  realtime-dev:
    image: 'supabase/realtime:v2.28.23'
    container_name: realtime-dev-q4s4w4k

Maybe the underlying problem is the format of "realtime-dev.supabase-realtime" (with the dot inbetween) and Coolify strips off everything after that dot when transforming? Or coolify checks if the service-name is string-included in the container_name and then uses the former. I don't know - probably something obvious to spot.

Anyway, here is the Kong log when trying to access realtime features via web socket:

2024/04/24 10:25:46 [error] 1193#0: *65269 [lua] init.lua:310: execute(): DNS resolution failed: dns server error: 3 name error. Tried: ["(short)realtime-dev.supabase-realtime:(na) - cache-miss","realtime-dev.supabase-realtime:1 - cache-hit/dns server error: 3 name error","realtime-dev.supabase-realtime:5 - cache-hit/dns server error: 3 name error"], client: 172.18.0.21, server: kong, request: "GET /realtime/v1/websocket?apikey=<...>&vsn=1.0.0 HTTP/1.1",

So, I believe when Coolify overrides the container-name, the name that kong expects (realtime-dev.supabase-realtime) to handle/proxx the realtime connections does not exist anymore.

Minimal Reproduction (if possible, example repository)

Install Supabase from Server Enable Realtime on any table and subscribe to the table from a client connected to that supabase instance

Exception or Error

Browser will show failed websocket connections, kong will show resolve errors:

2024/04/24 10:25:46 [error] 1193#0: *65269 [lua] init.lua:310: execute(): DNS resolution failed: dns server error: 3 name error. Tried: ["(short)realtime-dev.supabase-realtime:(na) - cache-miss","realtime-dev.supabase-realtime:1 - cache-hit/dns server error: 3 name error","realtime-dev.supabase-realtime:5 - cache-hit/dns server error: 3 name error"], client: 172.18.0.21, server: kong, request: "GET /realtime/v1/websocket?apikey=<...>&vsn=1.0.0 HTTP/1.1",

Version

v4.0.0-beta.265

[r3dm4n]

Facing same issue

[galacoder]

Facing same issue

[Geczy]

Can you change the container name it looks for by way of environment variable?

[actraiser]

Can you change the container name it looks for by way of environment variable?

The container name can not be edited in the environment variables, not by my knowledge. The correct name is in the docker-compose file, but it is being overriden by Coolify on deployment.

Greets -act

[actraiser]

by the way @andrasbacsai , there have been a couple of changes in the last days in the official Supabase docker-compose.yml that may or may not fix problems with the current Supabase template used in Coolify - like that queries in the SQL Editor will open up an error window and not save the query. Maybe you could update the Template by chance.

https://github.com/supabase/supabase/commits/master/docker/docker-compose.yml

Greets -act

[Geczy]

realtime works for me so does supabase sql does the realtime inspector work for you too or no?

[r3dm4n]

It doesn't work for me. I gave it up and am currently using Pocketbase, which does work in realtime

[actraiser]

realtime works for me so does supabase sql does the realtime inspector work for you too or no?

Could you please elaborate step-by-step how you got to a working Supabase (w/realtime) installation?

1. Which Coolify-Version?
2. After selecting Supabase as a new Resource from the coolify templates library, what env variables did you change?
3. Did you change any other settings in the stack (e.g. kong domain might be an obvious case or did you check "connect to pre-defined network", etc)

Please let me know, I will then reinstall the resource from scratch to validate if I can replicate a working installation.

Greets -act

[Geczy]

@actraiser pretty sure i just deployed the latest supabase yaml https://raw.githubusercontent.com/coollabsio/coolify/main/templates/compose/supabase.yaml

i also updated my environment variable

SERVICE_FQDN_SUPABASE_8000=https://supabase.dotabod.com

do you have the version from 2 weeks ago? i'm on v265 beta

[actraiser]

On Coolify v270, this unfortunately does not work - I know it worked at some point (maybe back in 265) but I just installed Supabase from scratch only changing the kong domain and and tested with both "connect to predefined network" checked and unchecked. Realtime will not working. Kong can not resolve the Realtime-Endpoint:

2024/04/30 06:13:44 [error] 1185#0: *77997 [lua] init.lua:310: execute(): DNS resolution failed: dns server error: 3 name error. Tried: ["(short)realtime-dev.supabase-realtime:(na) - cache-miss","realtime-dev.supabase-realtime:1 - cache-hit/dns server error: 3 name error","realtime-dev.supabase-realtime:5 - cache-hit/dns server error: 3 name error"], client: 172.18.0.3, server: kong, request: "GET /realtime/v1/websocket?apikey=<redacted>=1.0.0 HTTP/1.1", host: "<redacted>"

docker-compose in "Edit Docker Compose"-View (notice that the container_name is what Kong is looking for)

  realtime-dev:
    image: 'supabase/realtime:v2.28.23'
    container_name: realtime-dev.supabase-realtime

docker-compose in "Show Deployable Compose"-View (container_name has been renamed by coolify)

  realtime-dev:
    image: 'supabase/realtime:v2.28.23'
    container_name: realtime-dev-vs8coww

I hope @andrasbacsai has figureed out what to change in the docker-compose transformation to make this work again. I believe he posted in a reply on X some days ago, that realtime will be fixed.

Greets -act

[Geczy]

i just deployed latest supabase service using coolify v271

the url is resolved correctly for the realtime websocket

i do have a new issue that is solved with a new supabase commit that we don't have yet https://github.com/supabase/supabase/commit/03aa5c34740e63a7af09c46a749a4506985363dc

the error is:

Mixed Content: The page at 'https://supabasekong-xxxx.dotabod.com/project/default/realtime/inspector' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://supabasekong-xxxx.dotabod.com/realtime/v1/websocket?apikey=xxx.xxx.xxx-xxx&log_level=info&vsn=1.0.0'. This request has been blocked; this endpoint must be available over WSS.

[Geczy]

i updated the supabase service yaml here if you want to deploy it yourself it uses the new commits from supabase

i still have the wss issue, but i "solved" it by changing protocol: wss in the yaml. a new error presents itself in my admin dashboard though. but try this

https://gist.github.com/Geczy/fd7175a8a6479edf3637bb084a4d698a

[actraiser]

I am on v270 and I tried your generated template but Kong still cannot resolve the dns to the realtime service when it receives a websocket request. A quick way to test it without a front end application is to call the url :

https://kong-domain.com/realtime/v1/websocket?apikey=&vsn=1.0.0

which in my case returns the Kong-Error-Response from trying to resolve a realtime connection:

{
"message": "name resolution failed"
}

Kong's docker logs keep showing what I have already posted twice: DNS resolution failed: dns server error: 3 name error. Tried: ["(short)realtime-dev.supabase-realtime

Maybe its a completely different manner? I have "Connect to Predefined Network" checked in the Supabase stack, but I also tried it with de-selecting that option and connecting my client container manually into the SB stack network. This did NOT fix the realtime error. Could it be any other global Coolify setting? But then again, afair I did not customized anything other than setting my Coolify instance domain name.

I am sure it's something trivial but I can't figure it out. What is so frustrating, is, that it worked a couple of weeks ago and I am not sure what changed ever since.

Greets -act

[actraiser]

A solution that worked for me

cd /data/coolify/services/<resource_id_supabase_stack>/volumes/api
nano kong.yml

There I replaced "realtime-dev.supabase-realtime" with just 'realtime-dev', so it now looks like this:

  - name: realtime-v1
    _comment: 'Realtime: /realtime/v1/* -> wss://realtime-dev:4000/socket/*'
    url: http://realtime-dev:4000/socket/
      - name: realtime-v1-all
          - /realtime/v1/

After hitting "Pull latest Images & Restart" in Coolify - Realtime in Supabase works now!

I believe the underlying problem is still how Coolify deploys the docker-compose files and changes the container_name. The container name of service realtime_dev is supposed to be "realtime-dev.supabase-realtime" but as explained a couple of times, coolify changes it to "realtime-dev-[some-id]" - yet kong.yml is configured to proxy realtime to "realtime-dev.supabase-realtime" which does not exist then.

I would say, this is a bug with how Coolify treats the container_name fields of docker-compose files on deployment. What do you think @andrasbacsai ? I am not familiar enough with the internal workings to know for sure

Greets -act

[Geczy]

can't you do that in the kong.yml we bind here?

https://gist.github.com/Geczy/fd7175a8a6479edf3637bb084a4d698a#file-docker-compose-yaml-L33

[actraiser]

Oh yes, good catch - that is an even better place to implement the workaround fix.

Greets -act

[Geczy]

I am on v270 and I tried your generated template but Kong still cannot resolve the dns to the realtime service when it receives a websocket request. A quick way to test it without a front end application is to call the url :

kong-domain.com/realtime/v1/websocket?apikey=&vsn=1.0.0

i also get this error when hitting that endpoint, so maybe my realtime never worked afterall?

{
  "message": "name resolution failed"
}

if you go to /project/default/realtime/inspector can you subscribe to a channel? after some time i'm getting these errors:

{
  "channel": "3",
  "extension": "postgres_changes",
  "message": "Subscribing to PostgreSQL failed: {:error, \"Subscription insert failed with 0 rows. Check that tables are part of publication supabase_realtime and subscription params are correct: %{\\\"event\\\" => \\\"*\\\", \\\"schema\\\" => \\\"public\\\", \\\"table\\\" => \\\"*\\\"}\"}",
  "status": "error"
}

And accessing the realtime url i get HTTP ERROR 426

426 is not even in their list https://supabase.com/docs/guides/storage/debugging/error-codes

---

EDIT: ^^^ after adding a table to my database (where no tables existed), i'm now able to successfully subscribe to realtime changes via wss

however, the url you provided is still HTTP 426

some info reveals:

wget: server returned error: HTTP/1.1 426 Upgrade Required

so it looks like it's a protocol issue. now i'm trying this diff, confirmed the realtime inspector still works, it's grabbing wss successfully, despite setting protocol to ws

perhaps only websockets are supported, no more rest/http in supabase?

based on their readme, i don't even see http to be offered:

https://github.com/supabase/realtime/blob/6d5bb1eaba603ffcd341715c3331c3eb3c5fd8f4/README.md?plain=1#L154-L158

nothing in their code suggests http is valid:

https://github.com/search?q=realtime%2Fv1%2Fwebsocket&type=code

so in the end, these are the changes that work for me based on the new docker-compose file. and i updated my gist to reflect that, as well as the PR i have open #2101

596eaa7 (#2101)

[actraiser]

Sorry, I was not clear enough about the http-call - i just used it to quickly validate if Kong returns a 503 (name resolution failed in that case) or a 426 (can reach realtime server but does not authorize). Websocket is the only way to subscribe to Supabase realtime.

Greets -act

[znowfox]

lezgo andras !