Synapse 1.111.0 in Worker mode does not register new authenticated media endpoints

AlexanderBabel commented 3 months ago

Description

I use a workers configuration for my synapse with a media worker. I already updated my ingress to forward the new authenticated endpoints to the media worker instance. But all of them are returned with 404.

It looks like that the new endpoints are not registered on startup on the media server

Steps to reproduce

Use the 1.111.0 synapse version
Try to upload an image through the new endpoints/Retrieve images through the new endpoint

Homeserver

babel1.eu

Synapse Version

v1.111.0

Installation Method

Docker (matrixdotorg/synapse)

Database

postgres

Workers

Multiple workers

Platform

Ubuntu 22.04, Kubernetes

Configuration

general-config:
  enable_media_repo: false

media-worker-config:
  worker_app: "synapse.app.media_repository"

  worker_listeners:
    - port: 9090
      tls: false
      bind_addresses: ["::"]
      type: http

      resources:
        - names: [metrics]
          compress: false

    - port: 8083
      tls: false
      bind_addresses: ["::"]
      type: http
      x_forwarded: true

      resources:
        - names:
            - media
          compress: false

  worker_log_config: /synapse/config/log.yaml

Relevant log output

2024-07-23 13:52:49,139 - twisted - 282 - INFO - sentinel- SynapseSite starting on 9090
2024-07-23 13:52:49,139 - synapse.app._base - 439 - INFO - sentinel- Synapse now listening on TCP port 9090
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST /_matrix/media/v1/create
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST /_matrix/media/(r0|v3|v1)/upload$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for PUT /_matrix/media/v3/upload/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET /_matrix/media/(r0|v3|v1)/download/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)(/(?P<file_name>[^/]*))?$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET /_matrix/media/(r0|v3|v1)/thumbnail/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET /_matrix/media/(r0|v3|v1)/preview_url$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET /_matrix/media/(r0|v3|v1)/config$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/purge_media_cache$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/room/(?P<room_id>[^/]*)/media/quarantine$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/quarantine_media/(?P<room_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/quarantine/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/unquarantine/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/user/(?P<user_id>[^/]*)/media/quarantine$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/protect/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/unprotect/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET ^/_synapse/admin/v1/room/(?P<room_id>[^/]*)/media$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/delete$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/(?P<server_name>[^/]*)/delete$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for DELETE ^/_synapse/admin/v1/media/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET ^/_synapse/admin/v1/users/(?P<user_id>[^/]*)/media$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for DELETE ^/_synapse/admin/v1/users/(?P<user_id>[^/]*)/media$
2024-07-23 13:52:49,140 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.rest.health.HealthResource object at 0xe63816596b90> to path b'/health'
2024-07-23 13:52:49,141 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.rest.media.media_repository_resource.MediaRepositoryResource object at 0xe638165c30d0> to path b'/_matrix/media/r0'
2024-07-23 13:52:49,141 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.rest.media.media_repository_resource.MediaRepositoryResource object at 0xe638165c30d0> to path b'/_matrix/media/v3'
2024-07-23 13:52:49,141 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.rest.media.media_repository_resource.MediaRepositoryResource object at 0xe638165c30d0> to path b'/_matrix/media/v1'
2024-07-23 13:52:49,141 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.http.server.JsonResource object at 0xe63816368250> to path b'/_synapse/admin'
2024-07-23 13:52:49,141 - twisted - 282 - INFO - sentinel- SynapseSite starting on 8083
2024-07-23 13:52:49,141 - synapse.app._base - 439 - INFO - sentinel- Synapse now listening on TCP port 8083

2024-07-23 13:57:27,646 - synapse.access.http.8083 - 428 - DEBUG - GET-56- ::ffff: - 8083 - Received request: GET /_matrix/client/v1/media/download/babel1.eu/OHagbmtHsZRLwhrdDCGxRBOQ?allow_redirect=true
2024-07-23 13:57:27,647 - synapse.http.server - 130 - INFO - GET-56- <XForwardedForRequest at 0xe638160dc7d0 method='GET' uri='/_matrix/client/v1/media/download/babel1.eu/OHagbmtHsZRLwhrdDCGxRBOQ?allow_redirect=true' clientproto='HTTP/1.1' site='8083'> SynapseError: 404 - Unrecognized request
2024-07-23 13:57:27,648 - synapse.access.http.8083 - 473 - INFO - GET-56- ::ffff: - 8083 - {None} Processed request: 0.001sec/-0.000sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 59B 404 "GET /_matrix/client/v1/media/download/babel1.eu/OHagbmtHsZRLwhrdDCGxRBOQ?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) ElementNightly/2024072201 Chrome/126.0.6478.127 Electron/31.2.0 Safari/537.36" [0 dbevts]
2024-07-23 13:57:27,655 - synapse.replication.tcp.handler - 592 - DEBUG - process-replication-data-507- Received rdata presence (master) -> 78074332

Anything else that would be useful to know?

No response

tulir commented 3 months ago

The issue is that client and federation are not listed in resources, but I'm not sure if requiring them for the new endpoints is intentional or not. If it's intentional, it should probably be highlighted in release notes. If not, then it's a bug.

devonh commented 3 months ago

Agreed. Either we should:

update the docs to mention adding client and federation resources during worker configuration ...or
have the media resource also start the relevant client/federation servlets for the new endpoints

devonh commented 3 months ago

I think the preferred approach going forward should be to ensure that when specifying the media resource, all appropriate handlers are started in that situation.

4censord commented 1 month ago

It might be nice to add a link to this to the upgrade notes as well. As of right now, this is really confusing because it still does not work, even after modify the reverse proxy config as instructed, if you are deploying with a seperate media worker. (https://element-hq.github.io/synapse/latest/upgrade)

tulir commented 1 month ago

I don't think there's any point in adding upgrade notes for a bug that was already fixed and released 🤔

@devonh the issue can probably be closed now?

4censord commented 1 month ago

Ah, then I seem to have overlooked it in the release notes.

4censord commented 1 month ago

Just to clarify, in case someone else is confused with this: As of 1.114, this fix is included. 1.113 does not have it yet.

element-hq / synapse