search

element-hq / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://element-hq.github.io/synapse
GNU Affero General Public License v3.0
1.52k stars 186 forks source link

retry device resync doesn't follow exponentially back off algorithm #9603

Open matrixbot opened 10 months ago

matrixbot commented 10 months ago

This issue has been migrated from #9603.

Description

As far as I'm aware matrix-synapse should stop syncing to dead servers using an exponential back off algorithm. Reference: https://github.com/matrix-org/synapse/issues/4979 https://github.com/matrix-org/synapse/issues/5113

This actually doesn't happen if the error is "No Content-Type header received from remote server". Most likely the server owner disabled the server and left the reverse proxy online or the domain got resold and now there is another HTTP/HTTPS service on that domain.

In my case my server tries to contact it almost twice a minute and is spamming my logs. The exponential back off algorithm does not work in that case.

Logs in my case:

`2021-03-12 12:12:11,582 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1834 - {GET-O-15116} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:11,583 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1834 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:40,812 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1835 - {GET-O-15120} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:40,813 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1835 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:10,854 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1836 - {GET-O-15126} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:10,854 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1836 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:40,815 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1837 - {GET-O-15130} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:40,815 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1837 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:10,813 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1838 - {GET-O-15134} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:10,814 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1838 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:40,814 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1839 - {GET-O-15138} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:40,815 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1839 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:15:10,814 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1840 - {GET-O-15142} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:15:10,814 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1840 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server

Steps to reproduce

Version information

xundeenergie commented 1 month ago

My logfiles are filled with messages like

2024-09-10 08:07:42,972 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @orloskynedi:nyctico.de: 301: Moved Permanently
2024-09-10 08:07:42,994 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5059} [chat.paralilo.net] Request failed: GET matrix-federation://chat.paralilo.net/_matrix/federation/v1/user/devices/%40feliciaabebi%3Achat.paralilo.net: HttpResponseException('404: Not Found')
2024-09-10 08:07:42,994 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @feliciaabebi:chat.paralilo.net: 404: Not Found
2024-09-10 08:07:43,014 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5060} [nyctico.de] Request failed: GET matrix-federation://nyctico.de/_matrix/federation/v1/user/devices/%40glanville_luisa%3Anyctico.de: HttpResponseException('301: Moved Permanently')
2024-09-10 08:07:43,014 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @glanville_luisa:nyctico.de: 301: Moved Permanently
2024-09-10 08:07:43,037 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5061} [chat.paralilo.net] Request failed: GET matrix-federation://chat.paralilo.net/_matrix/federation/v1/user/devices/%40thadeusnepean%3Achat.paralilo.net: HttpResponseException('404: Not Found')
2024-09-10 08:07:43,037 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @thadeusnepean:chat.paralilo.net: 404: Not Found
2024-09-10 08:07:43,060 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5062} [nyctico.de] Request failed: GET matrix-federation://nyctico.de/_matrix/federation/v1/user/devices/%40grider_kellsie%3Anyctico.de: HttpResponseException('301: Moved Permanently')
2024-09-10 08:07:43,061 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @grider_kellsie:nyctico.de: 301: Moved Permanently
2024-09-10 08:07:43,082 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5063} [nyctico.de] Request failed: GET matrix-federation://nyctico.de/_matrix/federation/v1/user/devices/%40kaceygabriello%3Anyctico.de: HttpResponseException('301: Moved Permanently')

It starts every 20 to 30 seconds.

Version: matrix-synapse-py3 1.114.0+bookworm1

AndrewKvalheim commented 1 month ago

As an admin is there anything I can do manually to make Synapse forget these and stop trying?