search

matrix-org / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!
https://matrix-org.github.io/dendrite/
Apache License 2.0
5.67k stars 664 forks source link

storage: state NIDs missing from the database (0 != 1)" #1844

Closed grinapo closed 3 years ago

grinapo commented 3 years ago

Background information

Description

After git pull nothing works anymore, POSTs get 500 internal server errors

ERRO[2021-04-26T21:14:14.000494976Z] [github.com/matrix-org/dendrite/clientapi/routing/membership.go:385] checkMemberInRoom
         QueryCurrentState: could not query membership for user  error="storage: state NIDs missing from the database (0 != 1)" req.id=dlCjrEVVp0pJ req.method=POST req.path="/_matrix/client/r0/rooms/!yomrOFwgFXzmeMAbzX:matrix.org/read_markers" user_id="@grin:dendrite.grin.hu"
neilalexander commented 3 years ago

Maybe related to #1839.

neilalexander commented 3 years ago

If you open the room in question in Element and then re-issue another /join #room (ignoring the fact you are already joined), does this resolve the problem? This should rebuild the room state.

grinapo commented 3 years ago

No, 500 internal server error:

"0 : Error joining local room: \"QueryLatestEventsAndState: storage: state NIDs missing from the database (0 != 1)\""
johanhenselmans commented 3 years ago

Having the same problem: go version go1.16.3 linux/amd64 time="2021-05-03T13:26:14.353301969Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=5DbsqpvrKOcW req.method=POST req.path="/_matrix/client/r0/rooms/!MhiFxdeNqRVxQCBiQi:matrix.org/read_markers" user_id="@johanhenselmans:matrix.netsense.nl"

When trying to connect to #dendrite:matrix.org...

johanhenselmans commented 3 years ago

Perhaps and update of sqlite database, but not of psql database?

johanhenselmans commented 3 years ago

I noticed that in roomserver/storage/sqlite3/deltas/2021041615092700_state_blocks_refactor.go on line 146, in the same sequence in roomserver/storage/postgres/deltas/2021041615092700_state_blocks_refactor.go on line 195, the line maxsnapshotid++
is missing. Might that be the culprit?

zauberstuhl commented 3 years ago

Please let me know if you need more logging for debugging purposes. My system is not working at all because of that issue. Which is fine because only crazy people run on the master branch, still I am happy to help to resolve this as quickly as possible.

Thanks!

danpe commented 3 years ago

Same issue here 🙋🏻‍♂️

S7evinK commented 3 years ago

Based on the hint of @johanhenselmans: Anyone with a backup wants to try changing both occurrences of maxsnapshotid to newNID+1 here? https://github.com/matrix-org/dendrite/blob/1d89c4a3cd707773cf2c13c2579da856226f124e/roomserver/storage/postgres/deltas/2021041615092700_state_blocks_refactor.go#L195-L201

lantrix commented 3 years ago

I've hit the same thing on 0.3.11+1d89c4a3 and server unuseable. Happy to try to fix. @S7evinK I'll take a backup and try that.

lantrix commented 3 years ago

That didn't help.

Server logs before the change (I changed my domain and username in logs):

time="2021-05-15T04:15:41.344464254Z" level=info msg="Dendrite version 0.3.11+1d89c4a3" func=github.com/matrix-org/dendrite/setup.NewBaseDendrite file="github.com/matrix-org/dendrite/setup/base.go:112"
time="2021-05-15T04:15:42.038890470Z" level=info msg="Starting external Monolith listener on :8008" func="github.com/matrix-org/dendrite/setup.(*BaseDendrite).SetupAndServeHTTP.func2" file="github.com/matrix-org/dendrite/setup/base.go:427"
time="2021-05-15T04:15:51.915427437Z" level=info msg="Responding (174 bytes)" func="github.com/matrix-org/dendrite/mediaapi/routing.(*downloadRequest).jsonErrorResponse" file="github.com/matrix-org/dendrite/mediaapi/routing/download.go:160" MediaID=103cf667b75eb8bf1d0282b8c56e9817e3731d5eea3359dcf14ec51ff1e75ddf Origin=myserver code=404 req.id=GbgnjFe9rLmJ req.method=GET req.path=/_matrix/media/r0/thumbnail/myserver/103cf667b75eb8bf1d0282b8c56e9817e3731d5eea3359dcf14ec51ff1e75ddf
time="2021-05-15T04:15:52.012435123Z" level=info msg="Responding (174 bytes)" func="github.com/matrix-org/dendrite/mediaapi/routing.(*downloadRequest).jsonErrorResponse" file="github.com/matrix-org/dendrite/mediaapi/routing/download.go:160" MediaID=103cf667b75eb8bf1d0282b8c56e9817e3731d5eea3359dcf14ec51ff1e75ddf Origin=myserver code=404 req.id=Ihr1bqB0lumV req.method=GET req.path=/_matrix/media/r0/thumbnail/myserver/103cf667b75eb8bf1d0282b8c56e9817e3731d5eea3359dcf14ec51ff1e75ddf
time="2021-05-15T04:16:33.378017047Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=69oQBzaPaeDy req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-15T04:16:33.881259511Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=oeG37EN5fDxh req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-15T04:16:34.405791641Z" level=error msg="eventutil.BuildEvent failed" func=github.com/matrix-org/dendrite/clientapi/routing.generateSendEvent file="github.com/matrix-org/dendrite/clientapi/routing/sendevent.go:214" error="storage: state NIDs missing from the database (0 != 1)" req.id=e4yrYkX2MqHA req.method=PUT req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/send/m.room.message/m1621052005295.0" user_id="@lantrix:myserver"
time="2021-05-15T04:16:34.411787671Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=LsLdXixpYN71 req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-15T04:16:34.915118204Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=AbusO0AMx57P req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-15T04:16:35.427764433Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=LsY6FcOzaAnt req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-15T04:16:35.933924002Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=fwHZ44anFGgo req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-15T04:16:36.418102832Z" level=error msg="eventutil.BuildEvent failed" func=github.com/matrix-org/dendrite/clientapi/routing.generateSendEvent file="github.com/matrix-org/dendrite/clientapi/routing/sendevent.go:214" error="storage: state NIDs missing from the database (0 != 1)" req.id=Ri69RmbTZlWc req.method=PUT req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/send/m.room.message/m1621052005295.0" user_id="@lantrix:myserver"

After the change

time="2021-05-16T04:34:59.110727857Z" level=info msg="Dendrite version 0.3.11+24393208.1844-storage-state-nids-missing" func=github.com/matrix-org/dendrite/setup.NewBaseDendrite file="github.com/matrix-org/dendrite/setup/base.go:112"
time="2021-05-16T04:35:00.632758297Z" level=info msg="Starting external Monolith listener on :8008" func="github.com/matrix-org/dendrite/setup.(*BaseDendrite).SetupAndServeHTTP.func2" file="github.com/matrix-org/dendrite/setup/base.go:427"
time="2021-05-16T04:35:04.662555934Z" level=info msg="Responding (174 bytes)" func="github.com/matrix-org/dendrite/mediaapi/routing.(*downloadRequest).jsonErrorResponse" file="github.com/matrix-org/dendrite/mediaapi/routing/download.go:160" MediaID=103cf667b75eb8bf1d0282b8c56e9817e3731d5eea3359dcf14ec51ff1e75ddf Origin=myserver code=404 req.id=9VDrHN6D27NC req.method=GET req.path=/_matrix/media/r0/thumbnail/myserver/103cf667b75eb8bf1d0282b8c56e9817e3731d5eea3359dcf14ec51ff1e75ddf
time="2021-05-16T04:35:04.792176757Z" level=info msg="Responding (174 bytes)" func="github.com/matrix-org/dendrite/mediaapi/routing.(*downloadRequest).jsonErrorResponse" file="github.com/matrix-org/dendrite/mediaapi/routing/download.go:160" MediaID=103cf667b75eb8bf1d0282b8c56e9817e3731d5eea3359dcf14ec51ff1e75ddf Origin=myserver code=404 req.id=7NZRr4qIWd2v req.method=GET req.path=/_matrix/media/r0/thumbnail/myserver/103cf667b75eb8bf1d0282b8c56e9817e3731d5eea3359dcf14ec51ff1e75ddf
time="2021-05-16T04:35:13.634777507Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=gdOMeiHSEPZQ req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:14.101642653Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=amNR5xho1sxn req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:15.066742589Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=UE4DZMa6y0g1 req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:15.573708876Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=7dYugyzI4TPL req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:16.079201498Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=Iky5xaacyUpP req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:16.586969221Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=d1D4oV25cLaQ req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:17.093575333Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=NHEHpKt3nlpD req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:19.256593482Z" level=error msg="eventutil.BuildEvent failed" func=github.com/matrix-org/dendrite/clientapi/routing.generateSendEvent file="github.com/matrix-org/dendrite/clientapi/routing/sendevent.go:214" error="storage: state NIDs missing from the database (0 != 1)" req.id=306wxQexFGqs req.method=PUT req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/send/m.room.message/m1621052005295.0" user_id="@lantrix:myserver"
time="2021-05-16T04:35:19.465705384Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=yyHjUVRjNCtD req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:20.440211701Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=yOvpv60VGafc req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:20.945278217Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=swv3ef9pm5uj req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:21.269794478Z" level=error msg="eventutil.BuildEvent failed" func=github.com/matrix-org/dendrite/clientapi/routing.generateSendEvent file="github.com/matrix-org/dendrite/clientapi/routing/sendevent.go:214" error="storage: state NIDs missing from the database (0 != 1)" req.id=aMrw1qKLxW9M req.method=PUT req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/send/m.room.message/m1621052005295.0" user_id="@lantrix:myserver"
time="2021-05-16T04:35:21.494430705Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=AnCJf4AAlOCX req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:22.398666054Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=NVhC2ezSLcy9 req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:25.325590634Z" level=error msg="eventutil.BuildEvent failed" func=github.com/matrix-org/dendrite/clientapi/routing.generateSendEvent file="github.com/matrix-org/dendrite/clientapi/routing/sendevent.go:214" error="storage: state NIDs missing from the database (0 != 1)" req.id=xquSNAAKIcXZ req.method=PUT req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/send/m.room.message/m1621052005295.0" user_id="@lantrix:myserver"
time="2021-05-16T04:35:28.610094295Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=Wv7jBIjJH8a5 req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:33.368561365Z" level=error msg="eventutil.BuildEvent failed" func=github.com/matrix-org/dendrite/clientapi/routing.generateSendEvent file="github.com/matrix-org/dendrite/clientapi/routing/sendevent.go:214" error="storage: state NIDs missing from the database (0 != 1)" req.id=kr8lduna4Dh3 req.method=PUT req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/send/m.room.message/m1621052005295.0" user_id="@lantrix:myserver"
time="2021-05-16T04:35:44.165804076Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=WATTtPuU6WW7 req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/read_markers" user_id="@lantrix:myserver"
time="2021-05-16T04:35:49.384459404Z" level=error msg="eventutil.BuildEvent failed" func=github.com/matrix-org/dendrite/clientapi/routing.generateSendEvent file="github.com/matrix-org/dendrite/clientapi/routing/sendevent.go:214" error="storage: state NIDs missing from the database (0 != 1)" req.id=X7ZjSJ7mGChC req.method=PUT req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:myserver/send/m.room.message/m1621052005295.0" user_id="@lantrix:myserver"

Clients still unable to post to existing rooms.

lantrix commented 3 years ago

☝🏻 someone in the chat told me that this should be done with an empty DB (which I didn't do)

neilalexander commented 3 years ago

Just to document something that we discussed in #dendrite-dev — you might be able to un-break this by making Dendrite think you aren't in these rooms and then rejoining the room, forcing Dendrite to rebuild the room state.

This can be done by modifying the database:

  1. Look up your event_state_key_nid in the roomserver_event_state_key table
  2. Look up the room_nid in the roomserver_rooms table using the room ID
  3. Update the roomserver_membership table's membership_nid column from 3 to 1, matching the event_state_key_nid to the target_nid column and the room_nid to the room_nid column
  4. Try to rejoin the room from a client by doing a /join
zauberstuhl commented 3 years ago 
select event_state_key_nid from roomserver_event_state_keys where event_state_key = '@my:handle.com';
update roomserver_membership set membership_nid = 1 where target_nid = <event_state_key_nid> and membership_nid = 3;

@neilalexander that worked for me. is that still OK, even though I ignored the room ID comparison? I have a lot of rooms with the issue.

Thanks!

lantrix commented 3 years ago
  1. Look up your event_state_key_nid in the roomserver_event_state_key table
  2. Look up the room_nid in the roomserver_rooms table using the room ID
  3. Update the roomserver_membership table's membership_nid column from 3 to 1, matching the event_state_key_nid to the target_nid column and the room_nid to the room_nid column
  4. Try to rejoin the room from a client by doing a /join

Should I only have to rebuild this once per room, or once per user per room?

lantrix commented 3 years ago
  1. Look up your event_state_key_nid in the roomserver_event_state_key table
  2. Look up the room_nid in the roomserver_rooms table using the room ID
  3. Update the roomserver_membership table's membership_nid column from 3 to 1, matching the event_state_key_nid to the target_nid column and the room_nid to the room_nid column
  4. Try to rejoin the room from a client by doing a /join

Should I only have to rebuild this once per room, or once per user per room?

I did this for the 8 rooms I'm in and no luck.

_roomserver_event_statekey table:

select event_state_key_nid from roomserver_event_state_keys where event_state_key = '@lantrix:my.server';
 event_state_key_nid
---------------------
               65536
(1 row)

_roomserverrooms table:

select * from roomserver_rooms;
 room_nid |               room_id               | latest_event_nids | last_event_sent_nid | state_snapshot_nid | room_version
----------+-------------------------------------+-------------------+---------------------+--------------------+--------------
        6 | !nKPHVJZE1LyJSOQ0:my.server         | {576}             |                 576 |                282 | 6
        9 | !HjA8Bsu2XfwVm7Et:my.server         | {524}             |                 524 |                502 | 6
        8 | !TpQpEuJwaKGNaIABCD:matrix.org      | {563}             |                 563 |                545 | 6
        2 | !SABva3m49CGOrkkU:my.server         | {375}             |                 375 |                551 | 6
        3 | !jiFc2CzVELKyUjPv:my.server         | {577}             |                 577 |                336 | 6
        4 | !2ZA3XHpgL9cyUOOj:my.server         | {443}             |                 443 |                279 | 6
        5 | !0Ot8M3lZY1TYFlJI:my.server         | {444}             |                 444 |                281 | 6
        1 | !7FkQOIRRbaUNIB0P:my.server         | {495}             |                 495 |                255 | 6
        7 | !6EKuRjhwLKyo6bW8:my.server         | {397}             |                 397 |                261 | 6
(9 rows)

_roomservermembership table:

select * from roomserver_membership where target_nid = 65536;
 room_nid | target_nid | sender_nid | membership_nid | event_nid | target_local | forgotten
----------+------------+------------+----------------+-----------+--------------+-----------
        4 |      65536 |      65536 |              1 |       175 | t            | t
        5 |      65536 |      65536 |              1 |       312 | t            | f
        6 |      65536 |      65536 |              1 |       216 | t            | f
        3 |      65536 |      65536 |              1 |        84 | t            | f
        2 |      65536 |      65536 |              1 |        86 | t            | f
        1 |      65536 |      65536 |              1 |        85 | t            | f
        7 |      65536 |      65536 |              1 |       266 | t            | f
        9 |      65536 |      65536 |              1 |       315 | t            | f
(8 rows)

This is after I updated records with my target_nid and matching room_nid entries to membership_nid of 1 as suggested.

If I now form my client (element) am in the room with Internal room ID: !nKPHVJZE1LyJSOQ0:my.server

/join !nKPHVJZE1LyJSOQ0:my.server

I'm still getting the same error

time="2021-05-30T15:36:43.223660511Z" level=info msg="Dendrite version 0.3.11+3797d818" func=github.com/matrix-org/dendrite/setup.NewBaseDendrite file="github.com/matrix-org/dendrite/setup/base.go:112"
time="2021-05-30T15:36:44.083503842Z" level=info msg="Starting external Monolith listener on :8008" func="github.com/matrix-org/dendrite/setup.(*BaseDendrite).SetupAndServeHTTP.func2" file="github.com/matrix-org/dendrite/setup/base.go:427"
...
time="2021-05-30T15:34:24.011956627Z" level=error msg="QueryCurrentState: could not query membership for user" func=github.com/matrix-org/dendrite/clientapi/routing.checkMemberInRoom file="github.com/matrix-org/dendrite/clientapi/routing/membership.go:385" error="storage: state NIDs missing from the database (0 != 1)" req.id=C1TrfhgdhMOc req.method=POST req.path="/_matrix/client/r0/rooms/!nKPHVJZE1LyJSOQ0:my.server/read_markers" user_id="@lantrix:my.server

membership_nid stays at 1 in the database for this room_nid of 6 😞

lantrix commented 3 years ago

I'll see if I can debug the server and figure out what it's doing.

kegsay commented 3 years ago

Please can people confirm which database engine they are running when hitting this issue. I've identified a problem in the postgres migration only. Specifically, I'd like to hear from @danpe @johanhenselmans @Zauberstuhl

zauberstuhl commented 3 years ago

@kegsay PostgreSQL 12.5

danpe commented 3 years ago

@kegsay We used PostgreSQL 9.6 but now upgraded to PostgresQL 13.3

lantrix commented 3 years ago

I was running 13.1