Broken Invitation Links - http://kong is the domain

[iarata]

Bug report

• [x] I confirm this is a bug with Supabase, not with my own application.
• [x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

Currently, when sending an invitation to a user, the invitation link in the email is http://kong/auth/v1/verify?token=xxx rather than the proper domain.

To Reproduce

Steps to reproduce the behaviour, please provide code snippets or a repository:

1. Setup docker Supabase
2. Go to the authentication
3. Click "Add user"
4. Select "Send Invitation"
5. Check the invitation link in the email

Expected behaviour

It was expected that the invitation URL to contain the configured URLs in the .env file rather than http://kong as the domain.

[rmvh]

I think this was introduced in PR #999, where a bit of magic is used to determine the hostname regardless of the configured API_EXTERNAL_URL in the function isValidExternalHost here, although I'm not sure I understand it completely.

In my case the host in the confirmation mails is set to supabase-kong, which of course doesn't resolve outside of the docker network. It used to just use the API_EXTERNAL_URL correctly. This makes it difficult to test the sign-up flow during development.

[tomekit]

There is some breaking change in a minor release: v2.67.1 (https://github.com/supabase/gotrue/compare/v2.67.0...v2.67.1).

In our case, after upgrading Docker image to anything above v2.67.1 (including current latest) turns the account confirmation URL from https to https, there are also some other minor changes.

v2.67.0 (correct)

https://auth.domain.app/auth/v1/verify?token=token&type=signup&redirect_to=https://web.domain.app/login

v2.67.1 (broken)

http://auth.domain.app/auth/v1/verify?redirect_to=https%3A%2F%2Fweb.domain.app%2Flogin&token=token&type=signup

I've tried to reproduce this outside of the docker environment, kong etc. by injecting our configuration through .env, so I can run the debugger on main.go. I've set up breakpoints on: ConfirmationMail and isValidExternalHost and problem doesn't exist for some reason.

Unfortunately I don't have much time currently to add some additional logging to the code, compile the binary and debug/run it from the Docker context, to understand at which stage the URL does get broken.

[tomekit]

I suspect that it's part of the: isValidExternalHost function which unintentionally modifies the URL, specifically:

baseUrl := config.API.ExternalURL
    xForwardedHost := req.Header.Get("X-Forwarded-Host")
    xForwardedProto := req.Header.Get("X-Forwarded-Proto")
    if xForwardedHost != "" && xForwardedProto != "" {
        baseUrl = fmt.Sprintf("%s://%s", xForwardedProto, xForwardedHost)
    } else if req.URL.Scheme != "" && req.URL.Hostname() != "" {
        baseUrl = fmt.Sprintf("%s://%s", req.URL.Scheme, req.URL.Hostname())
    }

Sorry for mentioning you directly: @kangmingtay, but perhaps you could help us to understand the intention behind these two if/else cases where baseUrl (initially set from env) is overriden by the headers or request data.

In our case we run Kong in the HTTP context, as we terminate SSL at the upper layer. I suspect that because of Kong running in http context, it sets the: X-Forwarded-Proto (https://github.com/Kong/kong/issues/2202) header to HTTP and: isValidExternalHost function overrides the value specified in the API_EXTERNAL_URL header. I need to confirm that claim, but at this stage sounds plausible.

[tomekit]

Our infamous fix is here: https://github.com/supabase/auth/compare/v2.151.0...tomekit:auth:v2.151.0_tr it's likely not something that should be merged to supabase/gotrue, but it gives an idea where the problem is.

[kangmingtay]

@tomekit we use those headers to determine the hostname to use for email links and the baseUrl defaults to use the API_EXTERNAL_URL env var if those headers are missing.

In the supabase context, the API_EXTERNAL_URL is set to https://project_ref.supabase.co/auth/v1. With custom domains or vanity subdomains, the baseUrl can be either the custom domain, the vanity subdomain or the default domain assigned and it should be based on the hostname used in the request.

In our case we run Kong in the HTTP context, as we terminate SSL at the upper layer. I suspect that because of Kong running in http context, it sets the: X-Forwarded-Proto (https://github.com/Kong/kong/issues/2202) header to HTTP and: isValidExternalHost function overrides the value specified in the API_EXTERNAL_URL header. I need to confirm that claim, but at this stage sounds plausible.

does kong forward the original request url in a header?

[timkister]

I'm facing the same problem. My temporary fix is to switch to "supabase/gotrue:v2.67.0" in my docker-compose.yml

[tomekit]

does kong forward the original request url in a header?

I am not sure, I am using default kong setup from: https://supabase.com/docs/guides/self-hosting/docker
What header are we talking about? Would that be X-Forwarded-Path?

[donalffons]

I'm also having this issue (as reported in the original issue, with http://kong in the confirmation email) with a mostly default self-hosting docker-compose.yml + .env.

The issue is present with supabase/gotrue:v2.67.1 but goes away with supabase/gotrue:v2.67.0.

[EasternIndustries-IT]

I'm not sure if anyone has realized but in the docker-compose.yml it states SUPABASE_URL: http://kong/ changing this to use the API_EXTERNAL_URL variable should solve your issue. Or so one would think I have changed it to the local ip and i am getting it in the email link but I don't get the port. Which in my case is 8000 the api is supposed to be directed too.

[marcio-andrade]

I'm also having this issue (as reported in the original issue, with http://kong in the confirmation email) with a mostly default self-hosting docker-compose.yml + .env.

The issue is present with supabase/gotrue:v2.67.1 but goes away with supabase/gotrue:v2.67.0.

i verify and ok in 2.67.0 but using 2.99.00 not work correct

[marcio-andrade]

Is present bug in v2.67.1 and next. In v2.67.0 is correct use of SUPABASE_URL in sended e-mal link.

[philmas]

Issue is indeed still there. Would be nice to somehow be able to specify this.

[Obeyed]

How come this issue is not prioritized? If I'm not mistaken then all affected by this issue are vulnerable to man-in-the-middle attacks because of the unencrypted traffic. :thinking:

I'm self-hosting with docker and it seems that all email links are currently with http. It seems to be the case regardless of API_EXTERNAL_URL environment and any X-Forwarded-Proto headers.

Is there a workaround to force https for links in emails? What's the right way to get this up and running correctly?

Sorry for the direct mentions, but it seems that you guys could have some more info on this. @tomekit @kangmingtay Any help is appreciated, thanks!

[hf]

@Obeyed How are you self-hosting? What are the environment variables added to the GoTrue/Auth container?

[Obeyed]

With docker swarm behind nginx proxy. Have https context until my load balancer servers, from which the TLS is terminated and only http is used on the private network where kong, gotrue, and the remaining swarm runs.

Using image supabase/gotrue:v2.127.2

I assume these two environment variables are the relevant ones (not the real domains, but you get the idea):

API_EXTERNAL_URL: https://sub.domain1.com
GOTRUE_SITE_URL: https://sub.domain2.com

This is the example link from an invitation email that would be sent out:

http://sub.domain1.com/auth/v1/verify?token=some-token&type=invite&redirect_to=https://sub.domain2.com

Notice, the GOTRUE_SITE_URL (domain2) has the correct protocol, but the API_EXTERNAL_URL (domain1) uses the http protocol. I would assume that the entire API_EXTERNAL_URL including the protocol would be used. Is that not expected in some cases?

Any other info that can help?

[tomekit]

Hi @Obeyed, I am not sure if I can help here. In our case we're running fork with a couple amendments. Most importantly this line needs changing: https://github.com/supabase/gotrue/commit/c1f0ec8da6e8c19361789cb3d182a4c77c2763e5#diff-97dfc3310e5139b4d9c7a160bd077aeb81ca1e7b8d7cfa837a402f54b4c03a3dR213

Given that Supabase team is likely busy with other items your best bet is either to wait or apply it yourself and build a Docker image, alternatively use our build: s3drive/gotrue:v2.129.1 (https://hub.docker.com/layers/s3drive/gotrue/v2.129.1/images/sha256-a8176564a48f34b20598884848f284ed78cb8db0b5d44fea20d2ed8d37a4eab9?context=explore) which is the most recent Supabase version with these two fixes applied: https://github.com/supabase/gotrue/compare/master...tomekit:gotrue:master however please be aware that you shouldn't really use unofficial images for your organization... after all it's auth.

Good luck !

[Obeyed]

@hf do you need any more info? It would be great to get some insight into how this can be set up such that it works as expected. Right now I'm unsure of which HTTP Headers / Environment Variables to set to get the desired output with https. I have tried to set the HTTP Headers X-Forwarded-Host and X-Forwarded-Proto to force https via nginx through kong, but that didn't seem to affect the output. I'm assuming that kong forwards these headers. Haven't had time to dig deeper into this.

[hf]

Yes you probably need to set those headers deep in your stack, both in Kong and/or Nginx. GoTrue uses these headers to allow you to host the Auth server on multiple domains.

Unfortunately I'm not an expert in Kong so can't help you out there.

[hf]

If you want you can open a PR that adds a new config variable to ignore those headers in GoTrue.

[rmvh]

Up till version 2.67.0, this worked properly as expected. Though it's cool to be able to specify multiple custom domains (the feature in 2.67.1 which if I understand correctly introduced the breaking change), this leaves many of us self-hosters without the option to properly specify even a single domain. It seems weird to have an API_EXTERNAL_URL if it is going to be partially ignored, where the expectation is that that variable can be used to set - with absolute certainty and overriding all other values - the external URL for the API to be used anywhere, e.g. email templates.

If it is too much to ask for the original functionality to be brought back, perhaps the variable API_EXTERNAL_URL can be renamed to API_EXTERNAL_DOMAIN so that it is no longer misleading?

In the meantime, I thought I'd mention our solution which might be a bit obvious but could be useful to some: we've implemented custom endpoints for those actions that need to send emails (e.g. magic link, reset password), calling supabase.auth.admin.generateLink manually and then using javascript's URL to change the protocol and domain back to what it should be, and finally manually sending the mail. Doing it this way does require clients to call those endpoints instead of using the built-in Supabase methods, but it gives you full control over every aspect. See also this blog post about sending i18n-ready mails by Supabase's @mansueli

[dsyrstad]

This affects /recover as well. I self-host. You will never be to reliably determine the API URL from request headers. For example, I have my app and api fronted by AWS Cloudfront. Cloudfront terminates SSL and relays https://myapp.com/api/... requests to my kong instance, which then forwards to gotrue. You won't see "https://myapp.com" in any of the headers - ever. I also have API_EXTERNAL_URL set to https://myapp.com, but it has no effect with 2.132.0.

[lichong-a]

I also think that this problem is quite serious, and I hope it can be properly resolved, otherwise privatized deployment will be affected. If I have the ability I will also try to submit a PR.

[armandusbasson]

In the docker-compose.yml file there is a line SUPABASE_URL: http://kong Replace that with your project's URL, then stop and create the containers again:

// Stop and remove the containers
docker compose down

//Recreate and start the containers
docker compose up -d

This worked for me and the Invite emails are now giving the correct link.

[Chipsnet]

However, this means that STUDIO will attempt to access kong from the outside

For security reasons, it is recommended that studio access kong from within the docker container

While helpful in the short term, this problem needs to be fixed in the long term.

[Vokturz]

In my case it also happens when trying to reset my password :(

Any progress on this? I see this problem has been here for a while.

[nickmitchko]

This was bugging me hard. So I added a force no-change environment variable and submitted a PR.

With the changes:

1. Set API_EXTERNAL_URL environment variable as you normally would.
2. Add a new environment variable API_FORCE_EXTERNAL_URL to equal true.

.env:

API_EXTERNAL_URL=https://sb.mydomain.com/auth/v1
API_FORCE_EXTERNAL_URL=true

Kubernetes:

  environment:
    API_EXTERNAL_URL: https://sb.mydomain.com/auth/v1
    API_FORCE_EXTERNAL_URL: "true"

Fix repo: nickmitchko/auth Fix container (latest): nickmitchko/gotrue:latest Fix PR: #1571

@Vokturz, @Chipsnet, @hf

[Paillat-dev]

@nickmitchko Thanks much, because this was getting really boring...