Timeline filters messes up received room state

[PiotrKozimor]

Description

When state events are filtered out from timeline (in /sync response), state field still includes old events or does not include events at all. This is confusing as long as user does not get latest state of room (or does not get it at all).

Steps to reproduce

Have this script. Edit HS variable to that it points to fresh synapse (local would be the best, it creates room and puts state events to it). Run the script:

python3 -m pytest synapse_issue.py

The test case shall fail.

This test case passes on dendrite.

Version information

• Homeserver: self-hosted

• Version: 1.34

• Install method: official docker image

• Platform: k8s, container

[richvdh]

It looks like the filter is being set to

{"room": {"timeline": {"not_types":["my.custom.event"]}}}

@PiotrKozimor For clarity, can you show the sync response you get, and what you would expect?

[PiotrKozimor]

Yes, no problem. I have left only rooms field for clarity (others are not relevant). Extected response comes from dendrite, actual from synapse.

Expected

``` { "rooms": { "invite": {}, "join": { "!aZTcnTfR6Alyeps1:dendrite": { "account_data": { "events": [] }, "ephemeral": { "events": [] }, "state": { "events": [ { "content": { "foo": "grfuouzgqzviuj" }, "event_id": "$nnO6JoCEM9e236giMMVqcjG4JTF1xNhotOeOFWLIJfw", "origin_server_ts": 1624387876862, "sender": "@zcncsunwogrlyv:dendrite", "state_key": "one", "type": "my.custom.event", "unsigned": { "prev_content": { "foo": "ckqnycychntthy" }, "prev_sender": "@zcncsunwogrlyv:dendrite", "replaces_state": "$Snb87KMkje-LSv2u1gyn0d-ZuKXZonFn8Wm3NxLGjJE" } }, { "content": { "foo": "mnbhhzjpxkecqh" }, "event_id": "$Y1nfLP7g9QuNsbW6EdF6l7ilrsZMrw6o-9QmcBwyMvw", "origin_server_ts": 1624387876943, "sender": "@zcncsunwogrlyv:dendrite", "state_key": "two", "type": "my.custom.event", "unsigned": { "prev_content": { "foo": "zdcmzyrkpdsqaa" }, "prev_sender": "@zcncsunwogrlyv:dendrite", "replaces_state": "$vAcFSEx0o1I1VdJ007xlL2NtVS4rY6zi0CrIxO6y8jI" } }, { "content": { "foo": "gclqecjbstavcz" }, "event_id": "$BNyGed2A5GUwuWX5rdyRPrdHn732hTa1-Z6oJfsnts0", "origin_server_ts": 1624387877024, "sender": "@zcncsunwogrlyv:dendrite", "state_key": "three", "type": "my.custom.event", "unsigned": { "prev_content": { "foo": "iedxpoqshjioxu" }, "prev_sender": "@zcncsunwogrlyv:dendrite", "replaces_state": "$Lx8pLmR_3NGORxWfXDVkeSLknplDwMR-BpCGrHv4T-s" } } ] }, "timeline": { "events": [ { "content": { "creator": "@zcncsunwogrlyv:dendrite", "room_version": "6" }, "event_id": "$0uatKQUYDRAIxBqBNt8BXS1EDqJtygy-_Vba0Vr9-Z8", "origin_server_ts": 1624387874692, "sender": "@zcncsunwogrlyv:dendrite", "state_key": "", "type": "m.room.create" }, { "content": { "displayname": "zcncsunwogrlyv", "membership": "join" }, "event_id": "$e9m9qUPAs6HBY1viZr2tO0nW00HSC7zlsqpzHfGqAwE", "origin_server_ts": 1624387874692, "sender": "@zcncsunwogrlyv:dendrite", "state_key": "@zcncsunwogrlyv:dendrite", "type": "m.room.member" }, { "content": { "ban": 50, "events": { "m.room.aliases": 0, "m.room.avatar": 50, "m.room.canonical_alias": 50, "m.room.history_visibility": 100, "m.room.name": 50, "m.room.power_levels": 100 }, "events_default": 0, "invite": 0, "kick": 50, "notifications": { "room": 50 }, "redact": 50, "state_default": 50, "users": { "@zcncsunwogrlyv:dendrite": 100 }, "users_default": 0 }, "event_id": "$at7envb_kxr6dlX5jJUCFQQ9t7D6z2XTDMoUQEgT-6c", "origin_server_ts": 1624387874692, "sender": "@zcncsunwogrlyv:dendrite", "state_key": "", "type": "m.room.power_levels" }, { "content": { "join_rule": "invite" }, "event_id": "$DWq6ZgD7DSWsLt7XBiQYvzNSKtQ6jz7RZQizVXTnWyQ", "origin_server_ts": 1624387874692, "sender": "@zcncsunwogrlyv:dendrite", "state_key": "", "type": "m.room.join_rules" }, { "content": { "history_visibility": "shared" }, "event_id": "$R-5vjra24dMUfOfG0SEEbic6MKimoivFENqNBQjJDSc", "origin_server_ts": 1624387874692, "sender": "@zcncsunwogrlyv:dendrite", "state_key": "", "type": "m.room.history_visibility" } ], "limited": false, "prev_batch": "t1_1" } } }, "leave": {}, "peek": {} } } ```

Actual

```json { "rooms": { "invite": {}, "join": { "!syZVgKbGjBXuhlAzmb:dendrite": { "account_data": { "events": [] }, "ephemeral": { "events": [] }, "org.matrix.msc2654.unread_count": 0, "state": { "events": [] }, "summary": {}, "timeline": { "events": [ { "content": { "creator": "@zjangdesopjibi:dendrite", "room_version": "6" }, "event_id": "$fSssSjeYPxOHk27MNBtje8LxeNfoE5jtlg__bb6_3kU", "origin_server_ts": 1624387692833, "sender": "@zjangdesopjibi:dendrite", "state_key": "", "type": "m.room.create", "unsigned": { "age": 1921 } }, { "content": { "displayname": "zjangdesopjibi", "membership": "join" }, "event_id": "$L-KYnoDt4d4mToBZK4rO2zLDNkie-Ktgh_4uM6MrCkY", "origin_server_ts": 1624387692899, "sender": "@zjangdesopjibi:dendrite", "state_key": "@zjangdesopjibi:dendrite", "type": "m.room.member", "unsigned": { "age": 1855 } }, { "content": { "ban": 50, "events": { "m.room.avatar": 50, "m.room.canonical_alias": 50, "m.room.encryption": 100, "m.room.history_visibility": 100, "m.room.name": 50, "m.room.power_levels": 100, "m.room.server_acl": 100, "m.room.tombstone": 100 }, "events_default": 0, "invite": 50, "kick": 50, "redact": 50, "state_default": 50, "users": { "@zjangdesopjibi:dendrite": 100 }, "users_default": 0 }, "event_id": "$WQjk12jM5ElmniWNvkp3MbQiHomrNyl9EsCs565HgZc", "origin_server_ts": 1624387692943, "sender": "@zjangdesopjibi:dendrite", "state_key": "", "type": "m.room.power_levels", "unsigned": { "age": 1811 } }, { "content": { "join_rule": "public" }, "event_id": "$dC0CofzAFpH0KIXoquI_c3FASTVIZnqAN99sppD96us", "origin_server_ts": 1624387692978, "sender": "@zjangdesopjibi:dendrite", "state_key": "", "type": "m.room.join_rules", "unsigned": { "age": 1776 } }, { "content": { "history_visibility": "shared" }, "event_id": "$F9vwOBKEqglsLwCijxk1rCa1ge7tqGMN05n9P79UQNg", "origin_server_ts": 1624387693024, "sender": "@zjangdesopjibi:dendrite", "state_key": "", "type": "m.room.history_visibility", "unsigned": { "age": 1730 } } ], "limited": false, "prev_batch": "s36_2_0_1_1_1_1_2_1" }, "unread_notifications": { "highlight_count": 0, "notification_count": 0 } } }, "leave": {} } } ```

[richvdh]

It looks like dendrite's behaviour is incorrect here. https://matrix.org/docs/spec/client_server/r0.6.1#get-matrix-client-r0-sync says of state:

all state up to the start of the timeline, if since is not given.

(my emphasis).

We can see here that the timeline goes right back to the start of the room; the state at the start of the room is empty, which is why synapse returns an empty state.

(https://github.com/matrix-org/matrix-doc/issues/627 is an issue about the same question, from the spec side.)

[PiotrKozimor]

Thanks, I think I get the idea behind such behaviour looking at your comment. Do you want me to put ticket on dendrite for this?

[richvdh]

I think that would be good. Thanks!