Increase default `client_max_body_size` for federation requests

[davidmehren]

Describe the bug Incoming requests to /_matrix/federation/v1/send/ sometimes seem to be bigger than the default limit (matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb) of 50 megabytes. These get rejected by nginx, causing federation to fail.

Me and at least one other server had problems with incoming federation from matrix.org because of this, see https://github.com/matrix-org/synapse/issues/14492

To Reproduce I'm not sure how to reproduce this problem, as you probably need to have enough matrix.org rooms on your server to have the synapse over there try to send you federation requests bigger than 50 megabytes.

Expected behavior Federation works 😄

Matrix Server:

• OS: Ubuntu 20.04
• Architecture: amd64

[spantaleev]

matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb is defined in terms of matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb like this:

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/0365fea2fc1a3b481311ce9a9708d41536bec804/roles/custom/matrix-nginx-proxy/defaults/main.yml#L339

While matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb is hardcoded to 50 MB:

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/0365fea2fc1a3b481311ce9a9708d41536bec804/roles/custom/matrix-nginx-proxy/defaults/main.yml#L300-L301

---

So, matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb is effectively 150 MB, not 50 MB.

However, maybe matrix_synapse_max_upload_size_mb is hardcoded to 50 MB, but I'm not sure if the max_upload_size setting in homeserver.yaml makes Synapse itself reject larger federation requests. I believe that configuration setting is only used for media uploads.

So I expect that you're running with an effective federation max body of 150 MB by default. This seems generous enough, but.. let's try to learn more about the issue.

Judging by https://github.com/matrix-org/synapse/issues/14492, it appears that maybe the problem is not the max body size at all, but rather Synapse incorrectly (not generously enough) calculating the maximum request size.

Judging by https://github.com/matrix-org/synapse/commit/296a23f927447217fb529ad66ef82e66ab3d07ff, it seems like Synapse expects to receive (and is likely sending) 12.5 MB federation requests at the maximum. If so, a 50 MB max body size should be generous enough, and 150 MB (the playbook default) is overly so.

[davidmehren]

Weird.

The generated nginx config definitely contained a client_max_body_size of 50 MB and I first hand-edited this line in the generated config to 100M, which fixed the issue. This may be caused by our non-standard ansible-setup though, so I'll investigate.

it seems like Synapse expects to receive (and is likely sending) 12.5 MB federation requests at the maximum.

I have loads of log lines like matrix-nginx-proxy[1048723]: 2022/11/19 17:17:18 [error] 21#21: *20805917 client intended to send too large body: 61198322 bytes, client: 10.0.42.10, server: matrix.fachschaften.org, request: "PUT /_matrix/federation/v1/send/1668681485312 HTTP/1.1", host: "matrix.fachschaften.org:443" so something seems to be sending these huge federation requests. While I can't prove that it was matrix.org, it seems highly likely, as we have had federation problems with that server in the same timeframe these log lines appeared and at least one other server had problems too.

[davidmehren]

I think I figured it out:

We serve client and federation requests on the same port, as laid out in https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/0365fea2fc1a3b481311ce9a9708d41536bec804/docs/configuring-playbook-federation.md#changing-the-federation-port-from-8448-to-a-different-port-to-use-a-cdn-that-only-accepts-44380-ports. This leads to all requests using the matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb value, as the separate federation nginx templates with the bigger max_body_size value get completely ignored.

So in the end, the "use at your own risk" disclaimer in the docs was probably warranted. 😄 I could open a PR to add a hint to the docs, that increasing matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb when using that config may be a good idea? But maybe this is also some kind of Synapse bug, as federation requests > 50 MB seem quite large?

[spantaleev]

You could open a PR if you'd like!

Also, note that how we reverse-proxy to Synapse will change very soon, in an effort to address https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2090

But maybe this is also some kind of Synapse bug, as federation requests > 50 MB seem quite large?

I suppose you're right.. 50 MB federation requests sound oversized..