Remote media fetches fail from beeper.com since 1.110.0rc1

frebib commented 3 months ago

Description

Images never load, and the logs are spammed with these 429 errors

Steps to reproduce

Send an image from a beeper.com user to a user on a homeserver running Synapse 1.110.0rc1 or newer
Observe that the receiving party cannot view the image; it never loads and the homeserver logs are spammed with 429 errors.

Homeserver

nerdhouse.io, beeper.com

Synapse Version

1.110.0

Installation Method

Docker (matrixdotorg/synapse)

Database

postgres 15

Workers

Multiple workers

Platform

Docker on Linux

Configuration

presence: {enabled: true}

experimental_features:
  # for telegram ephemeral events
  msc2409_to_device_messages_enabled: true
  msc3202_transaction_extensions: true
  msc3202_device_masquerading: true
  msc3983_appservice_otk_claims: true
  msc3984_appservice_key_query: true

  msc3381_polls_enabled: true
  msc3026_enabled: true # busy presence state: https://github.com/matrix-org/matrix-doc/pull/3026
  msc3706_enabled: true # partial state in /send_join responses
  msc3773_enabled: true # thread notifications
  msc3664_enabled: true # pushrules to match on related events
  msc3981_enabled: true # recursive relations

I also have the LDAP module configured

Relevant log output

2024-07-03T16:40:09.488Z [WARN] synapse.http.matrixfederationclient:1496 - GET-230 - {GET-O-58} [local.beeper.com] Requested file size exceeds ratelimits
2024-07-03T16:40:09.488Z [WARN] synapse.media.media_repository:724 - GET-230 - Failed to fetch remote media local.beeper.com/frebib_xgkn8YR5945ryPvMU2FCBymR9iAbgFLD02RRRXKEeECqHq4fuNIArK1YMdgObjFY
2024-07-03T16:40:09.488Z [INFO] synapse.http.server:130 - GET-230 - <XForwardedForRequest at 0x7fe8c012b920 method='GET' uri='/_matrix/media/v3/download/local.beeper.com/frebib_xgkn8YR5945ryPvMU2FCBymR9iAbgFLD02RRRXKEeECqHq4fuNIArK1YMdgObjFY' clientproto='HTTP/1.0' site='unix'> SynapseError: 429 - Requested file size exceeds ratelimits
2024-07-03T16:40:09.489Z [INFO] synapse.access.http.unix:473 - GET-230 - 2a02:8010:64b4:3:b3fa:74bc:c162:2088 - unix - {None} Processed request: 0.245sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/1) 79B 429 "GET /_matrix/media/v3/download/local.beeper.com/frebib_xgkn8YR5945ryPvMU2FCBymR9iAbgFLD02RRRXKEeECqHq4fuNIArK1YMdgObjFY HTTP/1.0" "Element X/0.4.16 (Google Pixel 8 Pro; Android 14; AP2A.240605.024; Sdk 16aa6df0b)" [0 dbevts]

Anything else that would be useful to know?

There are also occasionally also authentication errors from Beeper's media store in S3, although I'm not sure if they're related. Not posting those logs because they're full of what look like keys

anoadragon453 commented 3 months ago

devonh commented 3 months ago

This looks like it is due to hitting this case (UNKNOWN_LENGTH of response): https://github.com/H-Shay/hq_synapse/blob/fa916558056013678e88d9dc2a2f64b161d9c77f/synapse/http/matrixfederationclient.py#L1495

In their config, max_upload_size was set to 1G and remote_media_download_burst_count was at the default of 500M. This would lead to the ratelimiter always rejecting any media request which didn't have a Content-Length header.

To state the problem more clearly, any homeserver which has max_upload_size configured higher than remote_media_download_burst_count will always fail downloading media from any source without a Content-Length header.

sumnerevans commented 2 months ago

We have deployed an update to the Beeper homeserver which should make it send the Content-Length header for downloads across federation.

turt2live commented 2 months ago

hah, I came by to say it looks like Beeper deployed a fix because I'm struggling to test my changes locally 😅

element-hq / synapse