mjohnson9
commented
2 years ago I'm experiencing the same issue. I have a nearly identical setup, but on Kubernetes. I'm using Cinny instead of Element, but I don't think that's relevant to this issue.
According to traffic graphs and logs, it seems that the bridges stopped receiving traffic at about 04:57 UTC on August 26th. I don't have any alerts from Prometheus indicating that I experienced any underlying faults at or around this time, nor was there any automated or manual maintenance activity happening. I can't find any events to correlate this to.
Here are the last 2 minutes of logs before the traffic to the bridges stopped. I may have been a bit overzealous with redactions, because I don't know what all of the values represent.
time="2022-08-26T04:55:51.873200708Z" level=info msg=PerformPusherCreation display_name="Element (iOS)" localpart=[REDACTED - ref: 18] pushkey="[REDACTED - ref: 4]" req.id=[REDACTED - ref: 0] req.method=POST req.path=/_matrix/client/r0/pushers/set user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:55:57.690995413Z" level=info msg="Uploading file" ContentType=image/jpeg FileSizeBytes=288124 Origin=johnson.gg UploadName=[REDACTED - ref: 2] req.id=[REDACTED - ref: 3] req.method=POST req.path=/_matrix/media/r0/upload user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:55:57.730133754Z" level=info msg="File uploaded" Base64Hash=[REDACTED - ref: 5] ContentType=image/jpeg FileSizeBytes=288124 Origin=johnson.gg UploadName=[REDACTED - ref: 2] media_id=[REDACTED - ref: 4] req.id=[REDACTED - ref: 1] req.method=POST req.path=/_matrix/media/r0/upload user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:55:57.861480910Z" level=info msg="Generated thumbnail" ActualHeight=32 ActualWidth=32 Height=32 Origin=johnson.gg ResizeMethod=crop Width=32 media_id=[REDACTED - ref: 4] processTime=13.699102ms req.id=[REDACTED - ref: 1] req.method=POST req.path=/_matrix/media/r0/upload user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:55:57.933846065Z" level=info msg="Signalling other goroutines waiting for this goroutine to generate the thumbnail." Height=32 Origin=johnson.gg ResizeMethod=crop Width=32 media_id=[REDACTED - ref: 4] req.id=[REDACTED - ref: 1] req.method=POST req.path=/_matrix/media/r0/upload user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:55:57.977970458Z" level=info msg="Generated thumbnail" ActualHeight=96 ActualWidth=96 Height=96 Origin=johnson.gg ResizeMethod=crop Width=96 media_id=[REDACTED - ref: 4] processTime=43.737987ms req.id=[REDACTED - ref: 1] req.method=POST req.path=/_matrix/media/r0/upload user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:55:57.981359692Z" level=info msg="Generated thumbnail" ActualHeight=720 ActualWidth=720 Height=720 MediaID=[REDACTED - ref: 4] Origin=johnson.gg ResizeMethod=scale Width=960 processTime=77.127208ms req.id=[REDACTED - ref: 6] req.method=GET req.path=/_matrix/media/r0/thumbnail/johnson.gg/[REDACTED - ref: 4]
time="2022-08-26T04:55:58.045096254Z" level=info msg="Signalling other goroutines waiting for this goroutine to generate the thumbnail." Height=96 Origin=johnson.gg ResizeMethod=crop Width=96 media_id=[REDACTED - ref: 4] req.id=[REDACTED - ref: 1] req.method=POST req.path=/_matrix/media/r0/upload user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:55:58.045208049Z" level=info msg="Signalling other goroutines waiting for this goroutine to generate the thumbnail." Height=720 MediaID=[REDACTED - ref: 4] Origin=johnson.gg ResizeMethod=scale Width=960 req.id=[REDACTED - ref: 6] req.method=GET req.path=/_matrix/media/r0/thumbnail/johnson.gg/[REDACTED - ref: 4]
time="2022-08-26T04:55:58.076225800Z" level=info msg="Generated thumbnail" ActualHeight=480 ActualWidth=480 Height=480 Origin=johnson.gg ResizeMethod=scale Width=640 media_id=[REDACTED - ref: 4] processTime=30.890368ms req.id=[REDACTED - ref: 1] req.method=POST req.path=/_matrix/media/r0/upload user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:55:58.094598595Z" level=info msg="Signalling other goroutines waiting for this goroutine to generate the thumbnail." Height=480 Origin=johnson.gg ResizeMethod=scale Width=640 media_id=[REDACTED - ref: 4] req.id=[REDACTED - ref: 1] req.method=POST req.path=/_matrix/media/r0/upload user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:55:58.241429640Z" level=info msg="Sent event to roomserver" event_id="$[REDACTED - ref: 12]" req.id=[REDACTED - ref: 7] req.method=PUT req.path="/_matrix/client/r0/rooms/![REDACTED - ref: 8]:johnson.gg/send/m.room.message/[REDACTED - ref: 9]" room_id="![REDACTED - ref: 8]:johnson.gg" room_version=9 user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:56:02.463263374Z" level=info msg="Sent event to roomserver" event_id="$[REDACTED - ref: 13]" req.id=[REDACTED - ref: 15] req.method=PUT req.path="/_matrix/client/r0/rooms/![REDACTED - ref: 8]:johnson.gg/send/m.room.message/[REDACTED - ref: 10]" room_id="![REDACTED - ref: 8]:johnson.gg" room_version=9 user_id="@[REDACTED - ref: 18]:johnson.gg"
time="2022-08-26T04:56:09.343720319Z" level=info msg="Sent event to roomserver" event_id="$[REDACTED - ref: 14]" req.id=[REDACTED - ref: 16] req.method=PUT req.path="/_matrix/client/v3/rooms/![REDACTED - ref: 8]:johnson.gg/send/m.room.message/[REDACTED - ref: 11]" room_id="![REDACTED - ref: 8]:johnson.gg" room_version=9 user_id="@[REDACTED - ref: 17]:johnson.gg"
time="2022-08-26T04:56:24.909898110Z" level=info msg="Sending EDU event" destinations=4115 edu_type=m.presence
MaximilianGaedig
S7evinK
neilalexander
Background information
go version: the one bundled with dockerDescription
Steps to reproduce
Unfortunatley after thorough investigation I could not find the cause nor a fix for this, everything looked normal, logs, database, logs of appservices/bridges, restarting does not fix the issue, restarted every docker container I had and it still persisted, this happened to another person in the matrix dendrite chat so this doesn't look like a faulty setup to me but an issue with 0.95. Downgrading to 0.9.4 did not help tho.
On request I can provide full logs but there is nothing to see there according to my investigations.
Edit: Thanks to @mjohnson9 for coming up with a workaround (deleting the jetstream directory)