search

element-hq / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://element-hq.github.io/synapse
GNU Affero General Public License v3.0
1.38k stars 161 forks source link

KeyError on recent background updates #16054

Open matrixbot opened 9 months ago

matrixbot commented 9 months ago

This issue has been migrated from #16054.

Description

We're seeing a KeyError on the recent updates introduced by https://github.com/matrix-org/synapse/commit/11e15d79b8a0af593fd9467e0cc7f8a9dfcb6c4f:

KeyError: 'current_state_events_stream_ordering_idx'
  File "synapse/storage/background_updates.py", line 405, in run_background_updates
    result = await self.do_next_background_update(sleep)
  File "synapse/storage/background_updates.py", line 528, in do_next_background_update
    update_info = self._background_update_handlers[self._current_background_update]

Specifically appears to be missing in BackgroundUpdates._background_update_handlers, but adding some logging indicates it is being registered as a background update:

{"message":"Registering background index update: current_state_events_stream_ordering_idx","namespace":"synapse.storage.background_updates","level":"info","time":1690998470.67,"request":"main","server_name":"beeper.com"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.9,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.97,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.97,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.98,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}

If I run locally with sqlite there is no error, but I can't see any codepath that would change the behaviour for postgres 🤔

schildbach commented 8 months ago

This database corruption issue could be a root cause: https://github.com/matrix-org/synapse/issues/13476