traefik bound to port 8449 also claims port 8448

[Kuchenmampfer]

Describe the bug After I updated the playbook and ran it, traefik does not want to start because it demands port 8448, which is already bound to my Caddy reverse proxy, in addition to port 8449.

To Reproduce The relevant part in my vars.yml file looks like this:

# the following two lines fix the port issue, but don't make it work because the federation tester wants to request from my IP instead of domain:
devture_traefik_config_entrypoint_web_secure_http3_enabled: false
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_http3_enabled: false

matrix_playbook_reverse_proxy_type: playbook-managed-traefik

# Ensure that public urls use https
matrix_playbook_ssl_enabled: true

# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval.
# This has the side-effect of also automatically disabling TLS for the matrix-federation entrypoint
# (by toggling `matrix_federation_traefik_entrypoint_tls`).
devture_traefik_config_entrypoint_web_secure_enabled: false

# If your reverse-proxy runs on another machine, consider using `0.0.0.0:81`, just `81` or `SOME_IP_ADDRESS_OF_THIS_MACHINE:81`
devture_traefik_container_web_host_bind_port: '127.0.0.1:81'

# We bind to `127.0.0.1` by default (see above), so trusting `X-Forwarded-*` headers from
# a reverse-proxy running on the local machine is safe enough.
# If you're publishing the port (`devture_traefik_container_web_host_bind_port` above) to a public network interface:
# - remove the `devture_traefik_config_entrypoint_web_forwardedHeaders_insecure` variable definition below
# - uncomment and adjust the `devture_traefik_config_entrypoint_web_forwardedHeaders_trustedIPs` line below
devture_traefik_config_entrypoint_web_forwardedHeaders_insecure: true
# devture_traefik_config_entrypoint_web_forwardedHeaders_trustedIPs: ['IP-ADDRESS-OF-YOUR-REVERSE-PROXY']

# Expose the federation entrypoint on a custom port (other than port 8448, which is normally used publicly).
#
# We bind to `127.0.0.1` by default (see above), so trusting `X-Forwarded-*` headers from
# a reverse-proxy running on the local machine is safe enough.
#
# If your reverse-proxy runs on another machine, consider:
# - using `0.0.0.0:8449`, just `8449` or `SOME_IP_ADDRESS_OF_THIS_MACHINE:8449` below
# - adjusting `matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_custom` (below) - removing `insecure: true` and enabling/configuring `trustedIPs`
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port: '127.0.0.1:8449'

# Depending on the value of `matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port` above,
# this may need to be reconfigured. See the comments above.
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_custom:
  forwardedHeaders:
    insecure: true
  # trustedIPs: ['IP-ADDRESS-OF-YOUR-REVERSE-PROXY']

Today, I updated the playbook and ran it to keep my server up to date. However, at the end (after 'wait a bit to let services start or fail'), several services failed, the first of which was traefik. So I checked its journal and saw that it complained about port 8448 being already bound to something else. That something else was my Caddy reverse proxy which was supposed to reverse proxy that port to traefik on port 8449. I disabled that part in Caddy and then traefik happily started. However, without my Caddy reverse Proxy, Federation did not work either. So I disabled HTTP/3. This solves the port issue, but now somehow the federation tester wants to request from my IP (https://152.53.21.223:8448/_matrix/key/v2/server) which does not work because Caddy only listens on the domain and I believe that is required for https to work.

Expected behavior traefic only binds port 8449 and not port 8448

Matrix Server:

• OS: Debian Bookworm
• Architecture: amd64

Additional context The relevant part of my Caddyfile:

kuchenmampfer.de {
    root * ...
    file_server

    reverse_proxy /.well-known/matrix/* https://matrix.kuchenmampfer.de {
                header_up Host {upstream_hostport}
    }
}

matrix.kuchenmampfer.de {
  handle {
        encode zstd gzip
        reverse_proxy localhost:81  {
               header_up X-Forwarded-Port {http.request.port}
               header_up X-Forwarded-TlsProto {tls_protocol}
               header_up X-Forwarded-TlsCipher {tls_cipher}
               header_up X-Forwarded-HttpsProto {proto}
        }
  }
}

matrix.kuchenmampfer.de:8448 {
    handle {
        encode zstd gzip
        reverse_proxy 127.0.0.1:8449 {
               header_up X-Forwarded-Port {http.request.port}
               header_up X-Forwarded-TlsProto {tls_protocol}
               header_up X-Forwarded-TlsCipher {tls_cipher}
               header_up X-Forwarded-HttpsProto {proto}
        }
    }
}

[Kuchenmampfer]

Looking into the related commit 9b5be68, I just found that there is a new variable matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port_udp and am now trying out what happens if I specifically set that to '127.0.0.1:8449'.

[Kuchenmampfer]

This also fixes the port issue. So I guess federationtester wanting to request from the IP is a separate issue.

For the port, I think the main question is whether that ..._udp variable should fall back to the regular variable or if it shall be added to Fronting the integrated reverse-proxy webserver with another reverse-proxy and be mentioned in the changelog.

[Kuchenmampfer]

I found a workaround for the IP issue by adding the following to my Caddyfile:

https://152.53.21.223:8448 {
    handle {
        redir https://matrix.kuchenmampfer.de:8448{uri}
    }
}

Its kinda hacky though, so I would be happy about suggestions where I might continue debugging.

[spantaleev]

Thank you for this investigation!

I've tried to add more documentation/examples about this in a56c2f8921bfaff6c31ef1669e14b04c022e386d.