Many unread and unreadable rooms existing since compression

matrix-org / rust-synapse-compress-state

A tool to compress some state in a Synapse instance's database

https://pypi.org/project/synapse-auto-compressor/

Apache License 2.0

143 stars 33 forks source link

Many unread and unreadable rooms existing since compression #27

Open telmich opened 3 years ago

telmich commented 3 years ago

After using https://github.com/matrix-org/rust-synapse-compress-state we see the following behaviour:

rooms that do not have new unread messages, are marked as unread
when selecting these rooms in element-web, the rooms do not show any content
When trying to access previous messages the error "failed to load timeline position" is raised

I am raising this issue here, as others running synapse might run into it.

I am currently trying to find out whether any kind of "matrix-db-fsck" exists that checks the integrity. Any pointers are welcome. I have raised the issue in the synapse-admins room as well.

clokep commented 3 years ago

I've had this moved to the compress repo since it seems related to running that tool.

telmich commented 3 years ago

Messages from the element-web console I see:

Could not find event $C-DUQVMF9fQJR4IVDL-jkgh9u9yJwT8EjTArFBp7JD8

In another room that exhibts this behaviour when trying to access a message by clicking "back on the timeline" button:

Error loading timeline panel at $zsMAnO1IfZj3a3IBCn1yl-yViFZll4vBh9zyHmd4llc: M_FORBIDDEN: You don't have permission to access that event.

telmich commented 3 years ago

Additionally some unencrypted rooms show notification symbols when there are no new messages.

clokep commented 3 years ago

It was pointed out that this might be matrix-org/synapse#6779, which is then made worse by using the state compressor. Maybe the query in https://github.com/matrix-org/synapse/issues/6779#issuecomment-582851569 could be run to see if that is related.

telmich commented 3 years ago

I just checked the query mentioned in https://github.com/matrix-org/synapse/issues/6779#issuecomment-582851569 and it does contain 0 rows.

What I see is that mostly 1 user account seems to be affected and a lot of private chats. Leaving a room and rejoining (even for unencrypted) does not restore the ability to see messages from before.

telmich commented 3 years ago

Something I also notice right now is that many of the unread/unreadable direct chats are actually listed below rooms instead of "people"

telmich commented 3 years ago

I was wondering whether anyone has a pointer to something like a "state fsck" that I could run to debug the situation?

telmich commented 2 years ago

It seems that this bug is still present in the current implementation. After continuing to apply the synapse_auto_compressor while synapse was offline, other members of the ungleich.ch homeserver reported afterwards the access denied to events.