error: 1 events exist in roomserver_events which have not been converted to a new state_snapshot_nid

matrixbot commented 2 weeks ago

This issue was originally created by @thomiel at https://github.com/matrix-org/dendrite/issues/2204.

Background information

Dendrite version or git SHA: 0.6.3
Monolith or Polylith?: Monolith
SQLite3 or Postgres?: SQLite
Running in Docker?: no
go version: 1.16 linux/arm
Client used (if applicable): Element

Description

What is the problem: can't connect to server
Who is affected:
How is this bug manifesting: server returns 503 response
When did this first appear: after upgrade from 0.3.x to 0.6.3

Steps to reproduce

pulled newest version from main branch
carefully made changes to the config
started the service

This is from the Monolith log:

time="2022-02-18T16:31:25.894137809Z" level=warning msg="State storage upgrade complete" func=github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas.UpStateBlocksRefactor file="github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas/2021041615092700_state_blocks_refactor.go:182"
time="2022-02-18T16:31:26.240958992Z" level=panic msg="failed to connect to room server db" func=github.com/matrix-org/dendrite/roomserver.NewInternalAPI file="github.com/matrix-org/dendrite/roomserver/roomserver.go:50" error="runDeltas: Failed run migration: ERROR 2021041615092700_state_blocks_refactor.go: failed to run Go migration function func(*sql.Tx) error: 1 events exist in roomserver_events which have not been converted to a new state_snapshot_nid; this is a bug, please report"
time="2022-02-18T16:31:28.496473588Z" level=info msg="Dendrite version 0.6.3+5106cc80" func=github.com/matrix-org/dendrite/setup/base.NewBaseDendrite file="github.com/matrix-org/dendrite/setup/base/base.go:126"
time="2022-02-18T16:31:28.523037385Z" level=warning msg="Performing state storage upgrade. Please wait, this may take some time!" func=github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas.UpStateBlocksRefactor file="github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas/2021041615092700_state_blocks_refactor.go:36"

matrixbot commented 2 weeks ago

This comment was originally posted by @neilalexander at https://github.com/matrix-org/dendrite/issues/2204#issuecomment-1046011188.

My feeling is that any events that are left without a converted snapshot after this migration should instead just have their snapshot NID set to 0 instead of panicking — then they stand a chance of being corrected by /get_missing_events, /state_ids etc when referenced in the future.

@kegsay — what do you think?

matrixbot commented 2 weeks ago

This comment was originally posted by @thomiel at https://github.com/matrix-org/dendrite/issues/2204#issuecomment-1046260080.

Is there something I can do to get my homeserver online again any time soon?

matrixbot commented 2 weeks ago

This comment was originally posted by @neilalexander at https://github.com/matrix-org/dendrite/issues/2204#issuecomment-1046792382.

Can you please try commit d138eb2 from #2209?

matrixbot commented 2 weeks ago

This comment was originally posted by @neilalexander at https://github.com/matrix-org/dendrite/issues/2204#issuecomment-1048009799.

Please retry with Dendrite 0.6.4.

matrixbot commented 2 weeks ago

This comment was originally posted by @thomiel at https://github.com/matrix-org/dendrite/issues/2204#issuecomment-1048245101.

I pulled the changes you merged into main yesterday (commit 280e9b19, one after Reset invalid state snapshots for events during state storage refactor migration #2209). The service repeats this block every half hour:

time="2022-02-22T20:49:50.589207424Z" level=warning msg="Performing state storage upgrade. Please wait, this may take some time!" func=github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas.UpStateBlocksRefactor file="github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas/2021041615092700_state_blocks_refactor.go:36"
time="2022-02-22T21:23:30.838585540Z" level=warning msg="State storage upgrade complete" func=github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas.UpStateBlocksRefactor file="github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas/2021041615092700_state_blocks_refactor.go:182"
time="2022-02-22T21:23:31.227850957Z" level=panic msg="failed to connect to room server db" func=github.com/matrix-org/dendrite/roomserver.NewInternalAPI file="github.com/matrix-org/dendrite/roomserver/roomserver.go:50" error="runDeltas: Failed run migration: ERROR 2021041615092700_state_blocks_refactor.go: failed to run Go migration function func(*sql.Tx) error: 1 events exist in roomserver_events which have not been converted to a new state_snapshot_nid; this is a bug, please report"
time="2022-02-22T21:23:33.653656634Z" level=info msg="Dendrite version 0.6.3+5106cc80" func=github.com/matrix-org/dendrite/setup/base.NewBaseDendrite file="github.com/matrix-org/dendrite/setup/base/base.go:126"

matrixbot commented 2 weeks ago

This comment was originally posted by @neilalexander at https://github.com/matrix-org/dendrite/issues/2204#issuecomment-1048258295.

Your log output suggests version 0.6.3+5106cc80 which is before the fix — are you sure you rebuilt?

In any case, v0.6.4 is out now and includes the PR so it may be easier to upgrade to the release version and try again.

matrixbot commented 2 weeks ago

This comment was originally posted by @thomiel at https://github.com/matrix-org/dendrite/issues/2204#issuecomment-1048818778.

Ah sorry, had too much contact with Python projects recently and forgot that Dendrite was done in a decent compilable language. Everything now works as expected again. Thanks!

element-hq / dendrite