spantaleev / matrix-docker-ansible-deploy

🐳 Matrix (An open network for secure, decentralized communication) server setup using Ansible and Docker
GNU Affero General Public License v3.0
4.74k stars 1.02k forks source link

Bridges cant sent media anymore #3451

Open wuast94 opened 1 month ago

wuast94 commented 1 month ago

Describe the bug When trying to send an image to a brdiged room i get ⚠ Your message may not have been bridged: failed to download media: failed to GET /_matrix/client/v1/media/download/[3ddruckbremen.de/c829d204cef7946e48bf2bb2fe7f427516d60e7b1817181090125709312](http://3ddruckbremen.de/c829d204cef7946e48bf2bb2fe7f427516d60e7b1817181090125709312): M_UNRECOGNIZED (HTTP 404): Unrecognized request

here are the logs of media repo:

Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.273 Z" level=warning msg="Continuing upload without lock! Set up Redis to make this warning go away." authUserId="@wuast94:3ddruckbremen.de" contentLength=23103 contentType=image/png filename=image.png host=matrix.3ddruckbremen.de method=POST queryString="filename=image.png" remoteAddr="172.24.0.2:51954" requestId=REQ-66 resource=/_matrix/media/r0/upload userAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.405 Z" level=info msg="Replying with result: *r0.MediaUploadedResponse &{ContentUri:mxc://3ddruckbremen.de/000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720}" contentLength=23103 contentType=image/png host=matrix.3ddruckbremen.de method=POST queryString="filename=image.png" remoteAddr="172.24.0.2:51954" requestId=REQ-66 resource=/_matrix/media/r0/upload userAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.495 Z" level=info msg="Replying with result: *_responses.DownloadResponse &{ContentType:image/png Filename:image.png SizeBytes:23103 Data:0xc0007462d0 TargetDisposition:infer}" contentLength=0 contentType="" host=matrix.3ddruckbremen.de method=GET queryString="" remoteAddr="172.24.0.2:51954" requestId=REQ-67 resource=/_matrix/media/r0/download/3ddruckbremen.de/000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720 userAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.705 Z" level=info msg="Replying with result: *_responses.DownloadResponse &{ContentType:image/png Filename:image.png SizeBytes:23103 Data:0xc0007463c0 TargetDisposition:infer}" contentLength=0 contentType="" host=matrix.3ddruckbremen.de method=GET queryString="" remoteAddr="172.24.0.2:51954" requestId=REQ-68 resource=/_matrix/media/r0/download/3ddruckbremen.de/000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720 userAgent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.812 Z" level=info msg="Fetching remote content..." authUserId="@wuast94:3ddruckbremen.de" contentLength=0 contentType="" host=matrix.3ddruckbremen.de method=GET queryString="ts=1722084300000&url=http%3A%2F%2F3ddruckbremen.de%2F000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720" remoteAddr="172.24.0.2:51954" requestId=REQ-69 resource=/_matrix/media/r0/preview_url userAgent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.814 Z" level=error msg="Error downloading content: Get \"http://3ddruckbremen.de/000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720\": host not allowed" authUserId="@wuast94:3ddruckbremen.de" contentLength=0 contentType="" host=matrix.3ddruckbremen.de method=GET queryString="ts=1722084300000&url=http%3A%2F%2F3ddruckbremen.de%2F000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720" remoteAddr="172.24.0.2:51954" requestId=REQ-69 resource=/_matrix/media/r0/preview_url userAgent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.878 Z" level=info msg="Replying with result: *_responses.ErrorResponse &{Code:M_NOT_FOUND Message:Not found InternalCode:M_NOT_FOUND}" contentLength=0 contentType="" host=matrix.3ddruckbremen.de method=GET queryString="ts=1722084300000&url=http%3A%2F%2F3ddruckbremen.de%2F000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720" remoteAddr="172.24.0.2:51954" requestId=REQ-69 resource=/_matrix/media/r0/preview_url userAgent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.879 Z" level=info msg="Replying with result: *_responses.ErrorResponse &{Code:M_NOT_FOUND Message:Not found InternalCode:M_NOT_FOUND}" contentLength=0 contentType="" host=matrix.3ddruckbremen.de method=GET queryString="ts=1722084300000&url=http%3A%2F%2F3ddruckbremen.de%2F000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720" remoteAddr="172.24.0.2:51974" requestId=REQ-72 resource=/_matrix/media/r0/preview_url userAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.879 Z" level=info msg="Replying with result: *_responses.DownloadResponse &{ContentType:image/png Filename:image.png SizeBytes:23103 Data:0xc0003238a8 TargetDisposition:infer}" contentLength=0 contentType="" host=matrix.3ddruckbremen.de method=GET queryString="height=600&method=scale&width=800" remoteAddr="172.24.0.2:51960" requestId=REQ-71 resource=/_matrix/media/r0/thumbnail/3ddruckbremen.de/000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720 userAgent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
Jul 27 12:45:17 matrix matrix-media-repo[2493183]: time="2024-07-27 12:45:17.880 Z" level=info msg="Replying with result: *_responses.DownloadResponse &{ContentType:image/png Filename:image.png SizeBytes:23103 Data:0xc0003238d8 TargetDisposition:infer}" contentLength=0 contentType="" host=matrix.3ddruckbremen.de method=GET queryString="height=600&method=scale&width=800" remoteAddr="172.24.0.2:51956" requestId=REQ-70 resource=/_matrix/media/r0/thumbnail/3ddruckbremen.de/000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720 userAgent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"

what wonders me in the error line is Error downloading content: Get \"http://3ddruckbremen.de/000d55052f2b43c52b3cfc83e81d17e18c6c6f4e1817179450475806720\": host not allowed"

i mean the url seems to be an internal one but the path is just a random string?

this happens on all bridges, telegram isnt even throughin an error. tried whatsapp telegram discord and instagram. sending in normal matrix rooms seems to work

To Reproduce matrix_mautrix_whatsapp_enabled: true matrix_media_repo_enabled: true

Expected behavior A clear and concise description of what you expected to happen.

Matrix Server:

spantaleev commented 1 month ago

For matrix-media-repo, we're exposing the new "authenticated media" APIs on Traefik's internal entrypoint, so that they're reachable at http://matrix-traefik:8008/_matrix/client/v1/...

That said, this does not seem to be related to the new "authenticated media" APIs, but to the old ones judging by your logs containing references to /_matrix/media/ (as opposed to /_matrix/client/..).

These old media routes have been handled like this (being captured at the internal Traefik entrypoint and being forwarded to matrix-media-repo) for a long time, so this is not something new.

I suppose the problem may be that matrix-media-repo insists on being called with the correct Host header (your homeserver name), not with matrix-traefik. As it says here:

Configuring MMR to use the correct server name can be challenging. MMR requires that it be approached with a Host header matching the name used in user IDs. This same server name must also be the name used in the config for the homeserver.

Note that the server name is before delegation. You may need to override the Host header at the reverse proxy before forwarding the request to MMR.


As a solution, I wonder if we could use Traefik's Headers middleware (and its customRequestHeaders option) for matrix-media-repo's internal Traefik entrypoint routers (in the labels file) to force-set the Host header correctly before the request is forwarded to matrix-media-repo.

You can try defining a new middleware and applying it to these routers. Example:

+# Perhaps matrix.3ddruckbremen.de should be used here, but you may also try 3ddruckbremen.de
+traefik.http.middlewares.matrix-media-repo-set-host-header.headers.customrequestheaders.Host=matrix.3ddruckbremen.de

-{% if middlewares | length > 0 %}
-traefik.http.routers.matrix-media-repo-internal-media.middlewares={{ middlewares | join(',') }}
-{% endif %}
+traefik.http.routers.matrix-media-repo-internal-media.middlewares=matrix-media-repo-set-host-header

If this helps, the labels.j2 template can be reworked to do things correctly.

I do not use matrix-media-repo myself, so I cannot test these changes.

wuast94 commented 1 month ago

i found this: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/ba04bace6db0fc4554c37970edd4989c5e05fd59/roles/custom/matrix-media-repo/defaults/main.yml#L96

which disables the old internal media completly if im right.

but no matter if i inculde or exclude the sub domain the error and logs are the same as above :/

i checked everytime if the config gets written to the labels file, and it does

spantaleev commented 1 month ago

This variable is overwritten via group_vars/matrix_servers and should effectively be set to true for you.

Regardless, it's likely the lack of a correct Host header that's tripping up matrix-media-repo.

wuast94 commented 1 month ago

also tested with X-Forwarded-Host but no luck sadly

wuast94 commented 1 month ago

i testet everything i could think of, also tried alot with chatgpt but no luck at all, i cant figure out what the problem is.

maybee if it helps this is what the whatsapp bridge reports: Jul 28 15:22:18 matrix matrix-mautrix-whatsapp[1346300]: Jul 28, 2024 15:22:18 ERR Error converting Matrix event error="failed to download media: failed to GET /_matrix/client/v1/media/download/3ddruckbremen.de/9fd5a0axxx5832576: M_UNRECOGNIZED (HTTP 404): Unrecognized request" action="handle matrix event" event_id=$Mst-HEpI_CDYVAxxx7BdALjg event_type=m.room.message portal_key=49165555555@s.whatsapp.net-49164444449@s.whatsapp.net room_id=!JCfxxxxxxSWYK:3ddruckbremen.de sender=@wuast94:3ddruckbremen.de

gnouts commented 3 days ago

Maybe offtopic but I was investigating that issue when I found that matrix_media_repo_container_labels_traefik_client_matrix_client_media_rule has | quote for PathRegexp, which ends up once deployed into something like :

# matrix/matrix-media-repo/labels
traefik.http.routers.matrix-media-repo-public-client-matrix-client-media.rule=Host(`matrix.domain.com`) && PathRegexp(`'/_matrix/client/(?P<version>(v1))/media'`)

with a double quote ( ` and ').

In my case, no paths were matching except for /_matrix, catched by "matrix-synapse-public-client-api@docker" and forwarded to synapse, instead of MMR.

Removing all | quote (for PathRegexp only!) in roles/custom/matrix-media-repo/defaults/main.yml fixed the images for all my clients. This also fixed my media for bridges.


I also had to follow again the setup with a proxy fronting traefik docs, especially that part, as I was still using devture_traefik_additional_entrypoints_auto. Unsure about the impact regarding this issue though.

spantaleev commented 3 days ago

Thanks for reporting this, @gnouts!

I've pushed this fix (to matrix-media-repo and a few other places) in https://github.com/spantaleev/matrix-docker-ansible-deploy/commit/05b79057aa62450c8638831e8d4a4a33138f11b8