SynapseError: 429 - Too Many Requests

llaera commented 2 years ago

Description

Hello everyone, I saw some issues on github talking about exactly this behaviour unfortunately when I try to invite people in a room it takes ages until synapse sends invites, I find the following errors in the logs.

2022-10-14 07:12:48,276 - synapse.http.server - 107 - INFO - POST-99625 - <XForwardedForRequest at 0x7f3693303af0 method='POST' uri='/_matrix/client/r0/rooms/!cUQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:12:53,621 - synapse.http.server - 107 - INFO - POST-99644 - <XForwardedForRequest at 0x7f3695cd1e80 method='POST' uri='/_matrix/client/r0/rooms/!cUaGXQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:00,212 - synapse.http.server - 107 - INFO - POST-99648 - <XForwardedForRequest at 0x7f369a39bbb0 method='POST' uri='/_matrix/client/r0/rooms/!cUaFQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:05,965 - synapse.http.server - 107 - INFO - POST-99654 - <XForwardedForRequest at 0x7f368d0cf9d0 method='POST' uri='/_matrix/client/r0/rooms/!cUaGdFQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:11,876 - synapse.http.server - 107 - INFO - POST-99667 - <XForwardedForRequest at 0x7f3695c848b0 method='POST' uri='/_matrix/client/r0/rooms/!cUaGXWdFQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:17,777 - synapse.http.server - 107 - INFO - POST-99671 - <XForwardedForRequest at 0x7f369bf7f6a0 method='POST' uri='/_matrix/client/r0/rooms/!cUaGXmtiOnU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:23,692 - synapse.http.server - 107 - INFO - POST-99678 - <XForwardedForRequest at 0x7f36995275e0 method='POST' uri='/_matrix/client/r0/rooms/!cUaG%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests

All the rooms and spaces are actually on our own server and we never invite people from other servers. I also tried to up the federation limit in the config with no change:

# The maximum number of concurrent federation requests allowed
# from a single server
federation_rc_reject_limit: 500

This started to happen when we created a lot of individual rooms for setting up our communication environment. I am using element.io as client and running the whole stack as docker containers behind the reverse proxy traefik 2, all on the latest version.

Is there any config I need to change in order for this to work?

Thank you in advance for any help. BR, Loris

Steps to reproduce

create a lot of rooms and inviting 3 people
no external server or user invitied

Homeserver

asemes.matrix.lu

Synapse Version

{"server":{"name":"Synapse","version":"1.68.0"}}

Installation Method

Docker (matrixdotorg/synapse)

Platform

Ubuntu 22.04.01 LTS 64 bit Dell R720 Client: Docker Engine - Community Version: 20.10.18 API version: 1.41 Go version: go1.18.6 Git commit: b40c2f6 Built: Thu Sep 8 23:11:43 2022 OS/Arch: linux/amd64 Context: default Experimental: true

Server: Docker Engine - Community Engine: Version: 20.10.18 API version: 1.41 (minimum version 1.12) Go version: go1.18.6 Git commit: e42327a Built: Thu Sep 8 23:09:30 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.8 GitCommit: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6 runc: Version: 1.1.4 GitCommit: v1.1.4-0-g5fd4c4d docker-init: Version: 0.19.0 GitCommit: de40ad0

Running in docker containers with postgres14, traefik 2.X, vectorim/element-web:latest and matrixdotorg/synapse:latest.

Relevant log output

2022-10-14 07:12:48,276 - synapse.http.server - 107 - INFO - POST-99625 - <XForwardedForRequest at 0x7f3693303af0 method='POST' uri='/_matrix/client/r0/rooms/!cUQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:12:53,621 - synapse.http.server - 107 - INFO - POST-99644 - <XForwardedForRequest at 0x7f3695cd1e80 method='POST' uri='/_matrix/client/r0/rooms/!cUaGXQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:00,212 - synapse.http.server - 107 - INFO - POST-99648 - <XForwardedForRequest at 0x7f369a39bbb0 method='POST' uri='/_matrix/client/r0/rooms/!cUaFQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:05,965 - synapse.http.server - 107 - INFO - POST-99654 - <XForwardedForRequest at 0x7f368d0cf9d0 method='POST' uri='/_matrix/client/r0/rooms/!cUaGdFQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:11,876 - synapse.http.server - 107 - INFO - POST-99667 - <XForwardedForRequest at 0x7f3695c848b0 method='POST' uri='/_matrix/client/r0/rooms/!cUaGXWdFQU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:17,777 - synapse.http.server - 107 - INFO - POST-99671 - <XForwardedForRequest at 0x7f369bf7f6a0 method='POST' uri='/_matrix/client/r0/rooms/!cUaGXmtiOnU%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests
2022-10-14 07:13:23,692 - synapse.http.server - 107 - INFO - POST-99678 - <XForwardedForRequest at 0x7f36995275e0 method='POST' uri='/_matrix/client/r0/rooms/!cUaG%3Amatrix.asemes.lu/invite' clientproto='HTTP/1.1' site='8008'> SynapseError: 429 - Too Many Requests



### Anything else that would be useful to know?

_No response_

DMRobertson commented 2 years ago

The easiest thing to do is to increase the rate limits for invites: see https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#rc_invites

If you're planning mass joins, you might also be aware of

All the rooms and spaces are actually on our own server and we never invite people from other servers. I also tried to up the federation limit in the config with no change:

The federation limit only affects communication between homeservers, so this won't have done anything.

DMRobertson commented 2 years ago

(It'd be great if the logs showed which rate limiter denied the request. Do we have an issue for that?)

salomaat commented 1 year ago

The easiest thing to do is to increase the rate limits for invites: see https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#rc_invites

If you're planning mass joins, you might also be aware of
* https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#rc_joins

* https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#rc_joins
All the rooms and spaces are actually on our own server and we never invite people from other servers. I also tried to up the federation limit in the config with no change:

The federation limit only affects communication between homeservers, so this won't have done anything.

matrix-org / synapse