search

element-hq / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!
https://element-hq.github.io/dendrite/
GNU Affero General Public License v3.0
33 stars 5 forks source link

Messages appear as sent but are not received anymore #2369

Closed matrixbot closed 3 weeks ago

matrixbot commented 3 weeks ago

This issue was originally created by @grisu48 at https://github.com/matrix-org/dendrite/issues/2369.

Background information

Description

Since yesterday I fail to receive any new messages on my previously flawlessly working dendrite 0.8.1 server. It worked fine until Friday, when suddenly I could not send/receive any new messages without any error or warning signs.

I can send messages, they appear as sent (the little checkbox appears) however those messages never appear on any other devices and/or users. Messages to public rooms get swallowed. Sending messages from a another user on the same instance appear sent but I never receive them.

All of my rooms (also the chatty ones) are now also quiet. The timestamp when the rooms stopped working varies - Some stopped receiving messages on Tuesday, I have others that worked nicely until Thursday.

When I send a message to one of my own rooms, this is what I could extract from the logs:

time="2022-04-23T07:00:19.959048235Z" level=info msg="lookupMissingStateViaStateIDs $LOIHHTwJdDCuUmubksAMTBL_NRNyz7FV4CSbLDG7Pmw" context=missing room_id="!CVxBQuGyZqlIJqTzuY:uhoreg.ca"
time="2022-04-23T07:00:20.399447692Z" level=info msg="Sent event to roomserver" event_id="$KfTy6yqsqOn9iKYUaZAH2g17LIVS8r-mMPBJLXHc7rw" req.id=9K4OFLrMQ4WR req.method=PUT req.path="/_matrix/client/r0/rooms/!ayPKSV6hofj2uZxI:matrix.feldspaten.org/send/m.room.encrypted/m1650697220197.2" room_id="!ayPKSV6hofj2uZxI:matrix.feldspaten.org" room_version=6 user_id="@phoenix:matrix.feldspaten.org"

I have no idea how I could debug this issue or provide some more meaningful insights. Help or input much appreciated. Thanks!

Steps to reproduce

No reproducer identified

matrixbot commented 3 weeks ago

This comment was originally posted by @grisu48 at https://github.com/matrix-org/dendrite/issues/2369#issuecomment-1109831486.

Update: Since yesterday I seem to be able to send/receive some messages. I still have a hole between when the system stopped and it recovered by itself but at least for now dendrite seems to be working again.

I haven't touched the server at all. The issue appeared out of the blue by itself and just as suddenly seems to go away after some days.

matrixbot commented 3 weeks ago

This comment was originally posted by @grisu48 at https://github.com/matrix-org/dendrite/issues/2369#issuecomment-1110237369.

I figured my instance was home to a lot of spam bots due to a misconfiguration on the recent schema change (open registration) - I guess my issue was only a simple consequence of this.

Closing this issue now as I had to nuke my whole instance to get rid of the infestation.