Most images don't load using 0.13.8

matrixbot commented 2 weeks ago

This issue was originally created by @makidoll at https://github.com/matrix-org/dendrite/issues/3425.

Background information

Dendrite version or git SHA: 0.13.8+763c79f
SQLite3 or Postgres?: Postgres (15-alpine)
Running in Docker?: Yes
go version: sh: go: not found
Client used (if applicable): Element on Web, Element X, Cinny

Description

What is the problem: Requested images return no data, but status 200 OK
Who is affected: Single user
How is this bug manifesting: Images rarely or never load in when I'm using the above clients
When did this first appear: Since the 0.13.8 update

I've tried resetting all my clients too. Example below returns Content-Length: 144 with Content-Type: application/json without data. Couldn't find any relevant logs either.

Steps to reproduce

Setup Dendrite monolith using Docker
Use any of the above clients

See images below

Screenshot from 2024-09-16 17-44-30 blurred

Screenshot from 2024-09-16 17-37-00 blurred

Screenshot from 2024-09-16 17-38-27 blurred

Screenshot from 2024-09-16 17-38-35

matrixbot commented 2 weeks ago

This comment was originally posted by @makidoll at https://github.com/matrix-org/dendrite/issues/3425#issuecomment-2354416197.

Seems like images might be coming through now on multiple devices. Maybe it's related to some sort of cache in Dendrite

matrixbot commented 2 weeks ago

This comment was originally posted by @makidoll at https://github.com/matrix-org/dendrite/issues/3425#issuecomment-2369935646.

This is still happening. Every time I restart any client, all images are lost but over time they start to appear. Very strange

matrixbot commented 2 weeks ago

This comment was originally posted by @makidoll at https://github.com/matrix-org/dendrite/issues/3425#issuecomment-2387250689.

https://github.com/matrix-org/dendrite/issues/3424 might have helped but still not fully fixed

matrixbot commented 2 weeks ago

This comment was originally posted by @makidoll at https://github.com/matrix-org/dendrite/issues/3425#issuecomment-2387273258.

Seems like the fix is to just refresh element a few times and the image requests slowly turn from content-type json to jpeg

matrixbot commented 2 weeks ago

This comment was originally posted by @S7evinK at https://github.com/matrix-org/dendrite/issues/3425#issuecomment-2387792935.

https://github.com/matrix-org/dendrite/blob/6cd1285ca0276bebd407110c37031bc4622f0b79/mediaapi/routing/routing.go#L204 Is the only place where we actively set the Content-Type to application/json. I couldn't find a reason why we would return HTTP 200 but still have application/json set, yet.

matrixbot commented 2 weeks ago

This comment was originally posted by @pat-s at https://github.com/matrix-org/dendrite/issues/3425#issuecomment-2395397173.

Since some weeks I also have this issue. Not exactly sure when it started, but reverting to 0.13.7 didn't help.

I see the following in the logs:

 time="2024-10-06T11:04:39.793349630Z" level=error msg="failed to open file" MediaID=VCJpgjatIIsIuTzgLKwjBiYx Origin=obermui.de error="os.Open: open /data/media_store/-/Q/YSg8KmsDhugLEHnnAShX7wSr2kgmuS │
│ 8BYCE7qJLWo/file: no such file or directory" req.id=1e7gVaofwHzQ req.method=OPTIONS req.path=/_matrix/media/v3/thumbnail/obermui.de/VCJpgjatIIsIuTzgLKwjBiYx

seems like dendrite is failing to query images from other servers? Did something change for synapse-based installations so that the path is now different?

For the above, the queried path doesn't seem to exist on the server:

https://obermui.de/_matrix/media/v3/thumbnail/obermui.de/VCJpgjatIIsIuTzgLKwjBiYx

EDIT: In my case it was likely a different issue and solved by https://github.com/matrix-org/dendrite/issues/2954#issuecomment-1407542336.

matrixbot commented 2 weeks ago

This comment was originally posted by @sleroq at https://github.com/matrix-org/dendrite/issues/3425#issuecomment-2418862533.

Seems like the fix is to just refresh element a few times and the image requests slowly turn from content-type json to jpeg

Seems like there is a limit on the number of thumbnails server able to generate at the same time. For me it's 5. When you refresh, browser won't download cached ones and eventually you load everything.

On the server it looks like this:

Details

``` monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | time="2024-10-17T08:08:23.240293449Z" level=info msg="Signalling other goroutines waiting for this goroutine to generate the thumbnail." Height=480 MediaID=a1039ef6a83a883e5579c65a2503482b6e10b878ccca5c189260a3fb76c60fbf Origin=sleroq.link ResizeMethod=scale Width=640 req.id=hobHFli7AMuA req.method=GET req.path=/_matrix/media/v3/thumbnail/sleroq.link/a1039ef6a83a883e5579c65a2503482b6e10b878ccca5c189260a3fb76c60fbf monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | time="2024-10-17T08:08:23.255353988Z" level=info msg="Signalling other goroutines waiting for this goroutine to generate the thumbnail." Height=480 MediaID=da2498ff0cd0c84686161db92cdbba24880e8a638a157b129b1b4ba5ddf4c9de Origin=sleroq.link ResizeMethod=scale Width=640 req.id=o7taqB4UG4ve req.method=GET req.path=/_matrix/media/v3/thumbnail/sleroq.link/da2498ff0cd0c84686161db92cdbba24880e8a638a157b129b1b4ba5ddf4c9de monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) monolith-1 | 2024/10/17 08:08:23 http: superfluous response.WriteHeader call from github.com/matrix-org/dendrite/mediaapi/routing.makeDownloadAPI.func1 (routing.go:214) ```

element-hq / dendrite