All requests returning 400 Bad Request after app rebuild

[michaelyork]

Hey Adam – thanks for the great work here!

Had been running my instance for weeks without issue, but interestingly upon upgrading dokku today and restarting/rebuilding all apps, my app is now returning a 400 on every request:

Nothing seems to be appearing in dokku's nginx access/error logs for the app either.

Any ideas?

[cooperaj]

I actually saw this one too. It looks like the update did something to Dokku's proxy ports setting. You'll need to reset them so that you're forwarding something like http:80:80 https:443:443.

[michaelyork]

I thought the same thing and verified / recreated those mappings but am getting the same response.

michaelyork@apps:~$ dokku proxy:report mastodon
=====> mastodon proxy information
       Proxy enabled:                 true
       Proxy port map:                http:80:80 https:443:443
       Proxy type:                    nginx

[cooperaj]

Mmm. I wonder if something has changed in the Linuxserver image.

[michaelyork]

Let me know if I can help debug somehow!

[cooperaj]

Have you managed to sort this? I've been playing around and can't see what would be going on. I've just updated to the new Mastodon release so maybe give that a shot?

[michaelyork]

Phew, figured it out! Was a conflict with this plugin and the custom nginx template you supply. Back up and running! Thanks for your help and your work on this project :)

[cooperaj]

Ah, yes. I don't think that plugin is necessary any more as the template (which is mostly stock where I can) has that value as configurable.

[Brunty]

Having just upgraded to 4.1.0 via this repo I'm now getting the same error, but have not got that plugin installed, did you take any steps that you can document for your debugging / fix @michaelyork?

Edit: dokku seemed to have disabled letsencrypt for the app, so I re-enabled that and it's not erroring with this, I'm getting a different error now!

If I then run dokku proxy:ports-set http:80:80 https:443:443 I go back to the first error...

[michaelyork]

Hey @Brunty – what's the full output of your git push to your dokku instance? I have a suspicion.

[Brunty]

@michaelyork can't provide that anymore as that terminal session is long gone.

I got it working again by various combinations of enabling let's encrypt and setting the proxy ports

I can try a force re-push later on if you'd like?

[michaelyork]

If you run into it again (or if any of us do on the next release 😄), let's dump those outputs here.

[cooperaj]

I'm now seeing this issue on my end. It may be slightly different but it looks like Dokku has somehow reverted my port settings to 80:80 443:80. Which of course is not working.

# dokku proxy:ports social.n8e.dev
-----> Port mappings for social.n8e.dev
    -----> scheme  host port  container port
    http           80         80
    https          443        80

I made no changes to cause this, it's happened overnight. I fixed it by running

# dokku proxy:ports-set social.n8e.dev http:80:80 https:443:443
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:80 https:443:443
-----> Overriding default nginx.conf with detected nginx.conf.sigil
-----> Configuring social.n8e.dev...(using app-supplied template)
-----> Creating https nginx.conf
       Enabling HSTS
       Reloading nginx

It looks like a letsencrypt job ran at just gone midnight

-----> social.n8e.dev needs renewal
=====> Enabling letsencrypt for social.n8e.dev
-----> Enabling ACME proxy for social.n8e.dev...
-----> Getting letsencrypt certificate for social.n8e.dev via HTTP-01
        - Domain 'social.n8e.dev'
2023/03/15 00:00:05 [INFO] [social.n8e.dev] acme: Obtaining bundled SAN certificate
2023/03/15 00:00:06 [INFO] [social.n8e.dev] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/210905496587
2023/03/15 00:00:06 [INFO] [social.n8e.dev] acme: Could not find solver for: tls-alpn-01
2023/03/15 00:00:06 [INFO] [social.n8e.dev] acme: use http-01 solver
2023/03/15 00:00:06 [INFO] [social.n8e.dev] acme: Trying to solve HTTP-01
2023/03/15 00:00:09 [INFO] [social.n8e.dev] The server validated our request
2023/03/15 00:00:09 [INFO] [social.n8e.dev] acme: Validations succeeded; requesting certificates
2023/03/15 00:00:10 [INFO] [social.n8e.dev] Server responded with a certificate.
-----> Certificate retrieved successfully.
-----> Installing let's encrypt certificates
-----> Unsetting DOKKU_PROXY_SSL_PORT
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:80
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:80 https:443:80
-----> Overriding default nginx.conf with detected nginx.conf.sigil
-----> Configuring social.n8e.dev...(using app-supplied template)
-----> Creating https nginx.conf
       Enabling HSTS
       Reloading nginx
-----> Ensuring network configuration is in sync for social.n8e.dev
-----> Overriding default nginx.conf with detected nginx.conf.sigil
-----> Configuring social.n8e.dev...(using app-supplied template)
-----> Creating https nginx.conf
       Enabling HSTS
       Reloading nginx
-----> Disabling ACME proxy for social.n8e.dev...
-----> Done

[michaelyork]

Confirmed: I just tried enabling letsencrypt on my install as well and same thing happened. Resetting proxy ports worked; I guess the LE plugin just always assumes those mappings.

I think this is a different issue than what I first experienced, though, as I had verified the ports were set correctly then.