Host: header not sent appropriately

kbrantley commented 5 years ago

I have a relatively simple config, where example.com/.well-known/matrix/server points to matrix.example.com:443, and the SRV record points to matrix.example.com:443 as well. In short, matrix connectivity for example.com is handled by matrix.example.com:443. homeserver.yaml has server_name configured as example.com as well.

This works! It works well. It works in the browser, it works in the desktop "app," and it works for the android app as well.

However, it doesn't work in the federation tester, likely because matrix.example.com:443 is a haproxy instance, which only passes requests where both the SNI parameter and the Host: header are set to matrix.example.com. Sample config:

    frontend matrix
            bind :::443 v4v6 ssl crt /etc/haproxy/certificates/matrix.example.com.pem alpn h2,http/1.1 strict-sni
            mode http
            tcp-request inspect-delay 5s
            http-response set-header Strict-Transport-Security "max-age=2592000; includeSubDomains; preload"
            option forwardfor
            http-request set-header X-Forwarded-Scheme https
            http-request del-header Connection

            acl host_matrix hdr(host) -i matrix.example.com
            acl host_service1 hdr(host) -i service1.matrix.example.com
            acl host_service2 hdr(host) -i service2.matrix.example.com

            acl authenticated http_auth(matrix)

            http-request auth realm matrix unless authenticated || host_matrix

            use_backend matrix if host_matrix
            use_backend service1 if host_service1
            use_backend service2 if host_service2

A way to summarize this config is "if the client is connecting to matrix.example.com, then use the matrix backend. There are some other services that should be used if they are requested. Lastly, require authentication for anything that isn't the matrix endpoint."

Likewise, what I'm seeing, is that the federation tester fails to verify that the federation is capable of working, and instead presets "msg=Failed to GET JSON to : <html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.[...]" which is a scenario where it very plainly isn't passing Host: matrix.example.com to the backend.

A potential workaround is for me to add a use_backend matrix line, which changes the default handler to be matrix when no other matching Host: header is found. When I do this, the federation tester works great and says that everything is just fine. However, this proves to me that the Host: header isn't being properly sent by the federation tester. It also means that any random time the IP address is accessed directly it'll be synapse responding, which is not what I want to have happen.

I performed a packet capture and was able to confirm that SNI is being sent properly, but logging the Host: header is proving to be annoying, so I don't have a 100% guarantee that it isn't being sent properly, aside from the behavior that I've described here.

It's possible that I'm misunderstanding the requirements for federation, but "clients trying to connect to matrix.example.com:443 should negotiate SNI and send an appropriate Host: header" seems like a rational thing to be doing.

Because I don't want to be "that person" who only provides sanitized/redacted configs and examples, my actual haproxy-fronted synapse instance is here. It's currently throwing a "no backend available" error for... the reasons described above.

Lastly, here is the output of the federation tester with the default backend configured, showing a success state: successful-federation.json.txt

turt2live commented 5 years ago

I think this is actually working correctly. From the spec:

If <delegated_hostname> is not an IP literal, and <delegated_port> is present, an IP address is discovered by looking up an AAAA or A record for <delegated_hostname>. The resulting IP address is used, alongside the <delegated_port>. Requests must be made with a Host header of <delegated_hostname>:<delegated_port>. The target server must present a valid certificate for <delegated_hostname>.

https://matrix.org/docs/spec/server_server/r0.1.2#resolving-server-names

ryanshow commented 4 years ago

@turt2live If I understand the problem correctly, the issue is that the tester is not sending a Host header along with the request. To quote from your quote:

Requests must be made with a Host header of :.

(emphasis mine)

So it sounds to me like when a request to a home server is made, a Host header must be present in the request. Else, how would a reverse proxy know where to forward requests to when it's proxying a bunch of domains on the same ports.

I'm running into a similar issue where evaluating whether federation is working over port 443 is proving difficult because the default site is swallowing the requests made by the federation tester.

richvdh commented 4 years ago

the federation tester does send a Host header as per the spec. Your problems lie elsewhere.

matrix-org / matrix-federation-tester

Host: header not sent appropriately #89