search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.82k stars 2.13k forks source link

sqlite3.OperationalError: no such table: event_txn_id_device_id when upgrading from Docker v1.89.0 to v1.92.2 #16335

Closed Anthial closed 11 months ago

Anthial commented 1 year ago

Description

Messages cannot be sent. When attempting to send a message the server logs a database error.

Steps to reproduce

Note, I do not currently have the capabilities to do this fresh, but assume a normal upgrade path from somewhere around v1.70.

  1. Run Synapse Docker v1.89.0 with sqlite3
  2. Upgrade to v1.92.2 via docker compose pull
  3. Messages can no longer be sent

Homeserver

the.eldritch.gift

Synapse Version

v1.92.2

Installation Method

Docker (matrixdotorg/synapse)

Database

SQLite (Docker Default)

Workers

Single process

Platform

Distro: Pop OS 22.04 Hardware:

Configuration

No response

Relevant log output

docker-synapse-1  | 2023-09-17 12:52:14,536 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 0/5
docker-synapse-1  | 2023-09-17 12:52:14,537 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 1/5
docker-synapse-1  | 2023-09-17 12:52:14,537 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 2/5
docker-synapse-1  | 2023-09-17 12:52:14,538 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 3/5
docker-synapse-1  | 2023-09-17 12:52:14,538 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 4/5
docker-synapse-1  | 2023-09-17 12:52:14,539 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 5/5
docker-synapse-1  | 2023-09-17 12:52:14,539 - synapse.http.server - 140 - ERROR - PUT-287 - Failed handle request via 'RoomSendEventRestServlet': <XForwardedForRequest at 0x7f35bb9654d0 method='PUT' uri='/_matrix/client/v3/rooms/!bMPYjpwMVArbXrMnrM%3Athe.eldritch.gift/send/m.room.encrypted/m1694955131659.2' clientproto='HTTP/1.1' site='8008'>
docker-synapse-1  | Traceback (most recent call last):
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
docker-synapse-1  |     result = context.run(
docker-synapse-1  |              ^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
docker-synapse-1  |     return g.throw(self.type, self.value, self.tb)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/rest/client/room.py", line 366, in _do
docker-synapse-1  |     ) = await self.event_creation_handler.create_and_send_nonmember_event(
docker-synapse-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 1012, in create_and_send_nonmember_event
docker-synapse-1  |     event = await self.get_event_from_transaction(
docker-synapse-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 931, in get_event_from_transaction
docker-synapse-1  |     existing_event_id = await self.get_event_id_from_transaction(
docker-synapse-1  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 901, in get_event_id_from_transaction
docker-synapse-1  |     await self.store.get_event_id_from_transaction_id_and_device_id(
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/events_worker.py", line 2031, in get_event_id_from_transaction_id_and_device_id
docker-synapse-1  |     return await self.db_pool.simple_select_one_onecol(
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1709, in simple_select_one_onecol
docker-synapse-1  |     return await self.runInteraction(
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 945, in runInteraction
docker-synapse-1  |     return await delay_cancellation(_runInteraction())
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
docker-synapse-1  |     result = context.run(
docker-synapse-1  |              ^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
docker-synapse-1  |     return g.throw(self.type, self.value, self.tb)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 911, in _runInteraction
docker-synapse-1  |     result = await self.runWithConnection(
docker-synapse-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1040, in runWithConnection
docker-synapse-1  |     return await make_deferred_yieldable(
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
docker-synapse-1  |     result = inContext.theWork()  # type: ignore[attr-defined]
docker-synapse-1  |              ^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
docker-synapse-1  |     inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
docker-synapse-1  |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
docker-synapse-1  |     return self.currentContext().callWithContext(ctx, func, *args, **kw)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
docker-synapse-1  |     return func(*args, **kw)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
docker-synapse-1  |     result = func(conn, *args, **kw)
docker-synapse-1  |              ^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1033, in inner_func
docker-synapse-1  |     return func(db_conn, *args, **kwargs)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 773, in new_transaction
docker-synapse-1  |     r = func(cursor, *args, **kwargs)
docker-synapse-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1752, in simple_select_one_onecol_txn
docker-synapse-1  |     ret = cls.simple_select_onecol_txn(
docker-synapse-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1775, in simple_select_onecol_txn
docker-synapse-1  |     txn.execute(sql, list(keyvalues.values()))
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 432, in execute
docker-synapse-1  |     self._do_execute(self.txn.execute, sql, parameters)
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 484, in _do_execute
docker-synapse-1  |     return func(sql, *args, **kwargs)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  | sqlite3.OperationalError: no such table: event_txn_id_device_id

Anything else that would be useful to know?

I do believe this was also the case for Synapse v1.90.0

michaelschefczyk commented 1 year ago

With postgresql, I did end up in a similar situation. I had to revert to 1.89.0, as I do not have docker images for the versions in between available. Before failing, the database was updated, so I was required to restore the previous version from a backup/snapshot.

clokep commented 1 year ago

Table was added in #15318, back in Synapse 1.83.0.

michaelschefczyk commented 1 year ago

Thank you very much @clokep.

Is there a practical way to add missing tables to the database? Would my homeserver otherwise be doomed and lost so that I would have to restart from zero under a different name - or use signal only instead of two alternatives?

clokep commented 1 year ago

Is there a practical way to add missing tables to the database? Would my homeserver otherwise be doomed and lost so that I would have to restart from zero under a different name - or use signal only instead of two alternatives?

I'm not sure how it missed the update to add it, but I haven't actually done any investigation.

Can you run SELECT * FROM applied_schema_deltas; and paste the results here?

michaelschefczyk commented 1 year ago

Sorry, we are tailing a different thread here. The similiarity is moving from v1.89.0 to v1.92.2 at least, but I am not absolutely certain about the rest. Should I file this as a separate topic??

The output is:

version | file
---------+---------------------------------------------------------------------------- 55 | 55/access_token_expiry.sql 55 | 55/track_threepid_validations.sql 55 | 55/users_alter_deactivated.sql 56 | 56/add_spans_to_device_lists.sql 56 | 56/current_state_events_membership.sql 56 | 56/current_state_events_membership_mk2.sql 56 | 56/delete_keys_from_deleted_backups.sql 56 | 56/destinations_failure_ts.sql 56 | 56/destinations_retry_interval_type.sql.postgres 56 | 56/device_stream_id_insert.sql 56 | 56/devices_last_seen.sql 56 | 56/drop_unused_event_tables.sql 56 | 56/event_expiry.sql 56 | 56/event_labels.sql 56 | 56/event_labels_background_update.sql 56 | 56/fix_room_keys_index.sql 56 | 56/hidden_devices.sql 56 | 56/nuke_empty_communities_from_db.sql 56 | 56/public_room_list_idx.sql 56 | 56/redaction_censor.sql 56 | 56/redaction_censor2.sql 56 | 56/redaction_censor3_fix_update.sql.postgres 56 | 56/redaction_censor4.sql 56 | 56/remove_tombstoned_rooms_from_directory.sql 56 | 56/room_key_etag.sql 56 | 56/room_membership_idx.sql 56 | 56/room_retention.sql 56 | 56/signing_keys.sql 56 | 56/signing_keys_nonunique_signatures.sql 56 | 56/state_group_room_idx.sql 56 | 56/stats_separated.sql 56 | 56/unique_user_filter_index.py 56 | 56/user_external_ids.sql 56 | 56/users_in_public_rooms_idx.sql 57 | 57/delete_old_current_state_events.sql 57 | 57/device_list_remote_cache_stale.sql 57 | 57/local_current_membership.py 57 | 57/remove_sent_outbound_pokes.sql 57 | 57/rooms_version_column.sql 57 | 57/rooms_version_column_2.sql.postgres 57 | 57/rooms_version_column_3.sql.postgres 58 | 58/00background_update_ordering.sql 58 | 58/02remove_dup_outbound_pokes.sql 58 | 58/03persist_ui_auth.sql 58 | 58/05cache_instance.sql.postgres 58 | 58/06dlols_unique_idx.py 58 | 58/07add_method_to_thumbnail_constraint.sql.postgres 58 | 58/07persist_ui_auth_ips.sql 58 | 58/08_media_safe_from_quarantine.sql.postgres 58 | 58/09shadow_ban.sql 58 | 58/10_pushrules_enabled_delete_obsolete.sql 58 | 58/10drop_local_rejections_stream.sql 58 | 58/10federation_pos_instance_name.sql 58 | 58/11user_id_seq.py 58 | 58/12room_stats.sql 58 | 58/13remove_presence_allow_inbound.sql 58 | 58/14events_instance_name.sql 58 | 58/14events_instance_name.sql.postgres 58 | 58/15_catchup_destination_rooms.sql 58 | 58/15unread_count.sql 58 | 58/16populate_stats_process_rooms_fix.sql 58 | 58/17_catchup_last_successful.sql 58 | 58/18stream_positions.sql 58 | 58/11dehydration.sql 58 | 58/11fallback.sql 58 | 58/19instance_map.sql.postgres 58 | 58/19txn_id.sql 58 | 58/20instance_name_event_tables.sql 58 | 58/20user_daily_visits.sql 58 | 58/21as_device_stream.sql 58 | 58/21drop_device_max_stream_id.sql 58 | 58/22puppet_token.sql 58 | 58/22users_have_local_media.sql 58 | 58/23e2e_cross_signing_keys_idx.sql 58 | 58/24drop_event_json_index.sql 58 | 58/25user_external_ids_user_id_idx.sql 58 | 58/26access_token_last_validated.sql 58 | 58/27local_invites.sql 58 | 58/28drop_last_used_column.sql.postgres 58 | 58/28rejected_events_metadata.sql 59 | 59/01ignored_user.py 59 | 59/02shard_send_to_device.sql 59 | 59/03shard_send_to_device_sequence.sql.postgres 59 | 59/04_event_auth_chains.sql 59 | 59/04_event_auth_chains.sql.postgres 59 | 59/04drop_account_data.sql 59 | 59/05cache_invalidation.sql 59 | 59/06chain_cover_index.sql 59 | 59/06shard_account_data.sql 59 | 59/06shard_account_data.sql.postgres 59 | 59/07shard_account_data_fix.sql 59 | 59/08delete_pushers_for_deactivated_accounts.sql 59 | 59/08delete_stale_pushers.sql 59 | 59/09rejected_events_metadata.sql 59 | 59/10delete_purged_chain_cover.sql 59 | 59/11drop_thumbnail_constraint.sql.postgres 59 | 59/12account_validity_token_used_ts_ms.sql 59 | 59/12presence_stream_instance.sql 59 | 59/12presence_stream_instance_seq.sql.postgres 59 | 59/13users_to_send_full_presence_to.sql 59 | 59/11add_knock_members_to_stats.sql 59 | 59/15locks.sql 59 | 59/16federation_inbound_staging.sql 59 | 59/14refresh_tokens.sql 60 | 60/01recreate_stream_ordering.sql.postgres 60 | 60/02change_stream_ordering_columns.sql.postgres 61 | 61/01change_appservices_txns.sql.postgres 61 | 61/01insertion_event_lookups.sql 61 | 61/02drop_redundant_room_depth_index.sql 61 | 61/02state_groups_state_n_distinct.sql.postgres 61 | 61/03recreate_min_depth.py 62 | 62/01insertion_event_extremities.sql 63 | 63/01create_registration_tokens.sql 63 | 63/02delete_unlinked_email_pushers.sql 63 | 63/02populate-rooms-creator.sql 63 | 63/03session_store.sql 63 | 63/04add_presence_stream_not_offline_index.sql 64 | 64/01msc2716_chunk_to_batch_rename.sql.postgres 65 | 65/01msc2716_insertion_event_edges.sql 65 | 65/02_thread_relations.sql 65 | 65/03remove_hidden_devices_from_device_inbox.sql 65 | 65/04_local_group_updates.sql 65 | 65/06remove_deleted_devices_from_device_inbox.sql 65 | 65/05_remove_room_stats_historical_and_user_stats_historical.sql 65 | 65/07_arbitrary_relations.sql 65 | 65/08_device_inbox_background_updates.sql 65 | 65/10_expirable_refresh_tokens.sql 65 | 65/11_devices_auth_provider_session.sql 67 | 67/01drop_public_room_list_stream.sql 68 | 68/01event_columns.sql 68 | 68/02_msc2409_add_device_id_appservice_stream_type.sql 68 | 68/03_delete_account_data_for_deactivated_accounts.sql 68 | 68/04_refresh_tokens_index_next_token_id.sql 68 | 68/04partial_state_rooms.sql 68 | 68/05partial_state_rooms_triggers.py 68 | 68/06_msc3202_add_device_list_appservice_stream_type.sql 69 | 69/01as_txn_seq.py 69 | 69/01device_list_oubound_by_room.sql 69 | 69/02cache_invalidation_index.sql 70 | 70/08_state_group_edges_unique.sql 70 | 70/01clean_table_purged_rooms.sql 71 | 71/01rebuild_event_edges.sql.postgres 71 | 71/01remove_noop_background_updates.sql 71 | 71/02event_push_summary_unique.sql 72 | 72/01add_room_type_to_state_stats.sql 72 | 72/01event_push_summary_receipt.sql 72 | 72/02event_push_actions_index.sql 72 | 72/03bg_populate_events_columns.py 72 | 72/03drop_event_reference_hashes.sql 72 | 72/03remove_groups.sql 72 | 72/04drop_column_application_services_state_last_txn.sql.postgres 72 | 72/05remove_unstable_private_read_receipts.sql 72 | 72/05receipts_event_stream_ordering.sql 72 | 72/06add_consent_ts_to_users.sql 72 | 72/06thread_notifications.sql 72 | 72/07force_update_current_state_events_membership.py 72 | 72/07thread_receipts.sql.postgres 72 | 72/08begin_cache_invalidation_seq_at_2.sql.postgres 72 | 72/08thread_receipts.sql 73 | 73/01event_failed_pull_attempts.sql 73 | 73/02add_pusher_enabled.sql 73 | 73/02room_id_indexes_for_purging.sql 73 | 73/03pusher_device_id.sql 73 | 73/03users_approved_column.sql 73 | 73/04partial_join_details.sql 73 | 73/04pending_device_list_updates.sql 73 | 73/05old_push_actions.sql.postgres 73 | 73/06thread_notifications_thread_id_idx.sql 73 | 73/08thread_receipts_non_null.sql.postgres 73 | 73/09partial_joined_via_destination.sql 73 | 73/09threads_table.sql 73 | 73/10_update_sqlite_fts4_tokenizer.py 73 | 73/10login_tokens.sql 73 | 73/11event_search_room_id_n_distinct.sql.postgres 73 | 73/12refactor_device_list_outbound_pokes.sql 73 | 73/13add_device_lists_index.sql 73 | 73/20_un_partial_stated_room_stream.sql 73 | 73/21_un_partial_stated_room_stream_seq.sql.postgres 73 | 73/22_rebuild_user_dir_stats.sql 73 | 73/22_un_partial_stated_event_stream.sql 73 | 73/23_fix_thread_index.sql 73 | 73/23_un_partial_stated_room_stream_seq.sql.postgres 73 | 73/24_events_jump_to_date_index.sql 73 | 73/25drop_presence.sql 74 | 74/01_user_directory_stale_remote_users.sql 74 | 74/90COMMENTS_destinations.sql.postgres 74 | 74/02_set_device_id_for_pushers_bg_update.sql 74 | 74/03_membership_tables_event_stream_ordering.sql.postgres 74 | 74/03_room_membership_index.sql 74 | 74/04_delete_e2e_backup_keys_for_deactivated_users.sql 74 | 74/04_membership_tables_event_stream_ordering_triggers.py 74 | 74/05_events_txn_id_device_id.sql 76 | 76/01_add_profiles_full_user_id_column.sql 76 | 76/02_add_user_filters_full_user_id_column.sql 76 | 76/03_per_user_experimental_features.sql 76 | 76/04_add_room_forgetter.sql 77 | 77/01_add_profiles_not_valid_check.sql.postgres 77 | 77/02_add_user_filters_not_valid_check.sql.postgres 77 | 77/03bg_populate_full_user_id_profiles.sql 77 | 77/04bg_populate_full_user_id_user_filters.sql 77 | 77/05thread_notifications_backfill.sql 77 | 77/06thread_notifications_not_null_event_push_actions.sql.postgres 77 | 77/06thread_notifications_not_null_event_push_actions_staging.sql.postgres 77 | 77/06thread_notifications_not_null_event_push_summary.sql.postgres 77 | 77/14bg_indices_event_stream_ordering.sql 78 | 78/01_validate_and_update_profiles.py 78 | 78/02_validate_and_update_user_filters.py 78 | 78/03_remove_unused_indexes_user_filters.py 78 | 78/03event_extremities_constraints.py 78 | 78/04_add_full_user_id_index_user_filters.py 78 | 78/04_read_write_locks_triggers.sql.postgres 79 | 79/03_read_write_locks_triggers.sql.postgres 79 | 79/04_mitigate_stream_ordering_update_race.py 79 | 79/05_read_write_locks_triggers.sql.postgres (214 lines)

What I wonder about is that after searching the huge log file, the beginning of troubles did look different, indeed. When starting the container after the upgrade, I did see output in the terminal which I took as an indication of the missing table. When I did roll back to 1.89.0, it said that the database version would be beyond 1.89.0, so that I had to scale back the database. When I look at the relevant time in the log file and then for the first exceptions, it does seem to start somewhat different - in an endless file:

2023-09-17 15:32:15,337 - synapse.storage.txn - 765 - WARNING - prune_old_user_ips-472439- [TXN OPERROR] {_prune_old_user_ips-2f58cd} server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. 0/5 2023-09-17 15:32:15,340 - twisted - 275 - CRITICAL - prune_old_user_ips-472439- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection result = func(conn, *args, *kw) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func return func(db_conn, args, **kwargs) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction conn.rollback() File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback self.conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:15,343 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:15,346 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback self._pool.disconnect(self._connection) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect raise Exception("wrong connection for thread") builtins.Exception: wrong connection for thread

2023-09-17 15:32:15,350 - synapse.metrics.background_process_metrics - 244 - ERROR - prune_old_user_ips-472439- Background process 'prune_old_user_ips' threw an exception Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction r = func(cursor, *args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/client_ips.py", line 484, in _prune_old_user_ips_txn txn.execute(sql, (timestamp,)) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute self._do_execute(self.txn.execute, sql, parameters) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute return func(sql, args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ psycopg2.OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/synapse/metrics/background_process_metrics.py", line 242, in run return await func(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/client_ips.py", line 486, in _prune_old_user_ips await self.db_pool.runInteraction( File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction return await delay_cancellation(_runInteraction()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction result = await self.runWithConnection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection return await make_deferred_yieldable( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, kw) ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection result = func(conn, *args, kw) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func return func(db_conn, *args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction conn.rollback() File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback self.conn.rollback() File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 66, in rollback raise ConnectionLost() twisted.enterprise.adbapi.ConnectionLost 2023-09-17 15:32:16,147 - synapse.storage.txn - 765 - WARNING - StoreKeyFetcher-322756- [TXN OPERROR] {get_server_keys_json-2f58ce} server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. 0/5 2023-09-17 15:32:16,150 - twisted - 275 - CRITICAL - StoreKeyFetcher-322756- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection result = func(conn, args, kw) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction conn.rollback() File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback self.conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:16,153 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:16,156 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback self._pool.disconnect(self._connection) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect raise Exception("wrong connection for thread") builtins.Exception: wrong connection for thread

2023-09-17 15:32:16,162 - synapse.crypto.keyring - 687 - INFO - PerspectivesKeyFetcher-80445- Requesting keys [_FetchKeyRequest(server_name='vac.systemec.nl', minimum_valid_until_ts=1694957536141, key_ids=['ed25519:a_MLcI'])] from notary server matrix.org 2023-09-17 15:32:17,194 - synapse.http.matrixfederationclient - 348 - INFO - PerspectivesKeyFetcher-80445- {POST-O-650059} [matrix.org] Completed request: 200 OK in 1.03 secs, got 450 bytes - POST matrix-federation://matrix.org/_matrix/key/v2/query 2023-09-17 15:32:17,201 - synapse.storage.txn - 765 - WARNING - PerspectivesKeyFetcher-80445- [TXN OPERROR] {store_server_keys_json-2f58cf} server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. 0/5 2023-09-17 15:32:17,204 - twisted - 275 - CRITICAL - PerspectivesKeyFetcher-80445- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection result = func(conn, *args, *kw) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func return func(db_conn, args, **kwargs) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction conn.rollback() File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback self.conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:17,208 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:17,210 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback self._pool.disconnect(self._connection) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect raise Exception("wrong connection for thread") builtins.Exception: wrong connection for thread

2023-09-17 15:32:17,215 - synapse.crypto.keyring - 646 - ERROR - PerspectivesKeyFetcher-80445- Unable to get key from 'matrix.org': InterfaceError connection already closed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction r = func(cursor, *args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn return self.simple_upsert_txn_native_upsert( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert txn.execute(sql, list(allvalues.values())) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute self._do_execute(self.txn.execute, sql, parameters) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute return func(sql, args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ psycopg2.OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func return func(db_conn, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction conn.rollback() File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback self.conn.rollback() File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 66, in rollback raise ConnectionLost() twisted.enterprise.adbapi.ConnectionLost

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 638, in get_key return await self.get_server_verify_key_v2_indirect( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 741, in get_server_verify_key_v2_indirect processed_response = await self.process_v2_response( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 601, in process_v2_response await make_deferred_yieldable( File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/keys.py", line 170, in store_server_keys_json await self.db_pool.simple_upsert( File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1243, in simple_upsert return await self.runInteraction( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction return await delay_cancellation(_runInteraction()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction result = await self.runWithConnection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection return await make_deferred_yieldable( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(*args, *kw) ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection result = func(conn, args, kw) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1021, in inner_func self.engine.attempt_to_set_autocommit(conn, False) File "/usr/local/lib/python3.11/site-packages/synapse/storage/engines/postgres.py", line 220, in attempt_to_set_autocommit return conn.set_session(autocommit=autocommit) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ psycopg2.InterfaceError: connection already closed 2023-09-17 15:32:17,234 - synapse.http.federation.well_known_resolver - 253 - INFO - ServerKeyFetcher-112- Fetching https://vac.systemec.nl/.well-known/matrix/server 2023-09-17 15:32:17,380 - synapse.http.federation.well_known_resolver - 190 - INFO - ServerKeyFetcher-112- Response from .well-known: {'m.server': 'vac.systemec.nl:443'} 2023-09-17 15:32:17,543 - synapse.http.matrixfederationclient - 348 - INFO - ServerKeyFetcher-112- {GET-O-650060} [vac.systemec.nl] Completed request: 200 OK in 0.31 secs, got 313 bytes - GET matrix-federation://vac.systemec.nl/_matrix/key/v2/server 2023-09-17 15:32:17,550 - synapse.storage.txn - 765 - WARNING - ServerKeyFetcher-112- [TXN OPERROR] {store_server_keys_json-2f58d0} server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. 0/5 2023-09-17 15:32:17,552 - twisted - 275 - CRITICAL - ServerKeyFetcher-112- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection result = func(conn, *args, *kw) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func return func(db_conn, args, **kwargs) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction conn.rollback() File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback self.conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:17,556 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:17,558 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback self._pool.disconnect(self._connection) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect raise Exception("wrong connection for thread") builtins.Exception: wrong connection for thread

2023-09-17 15:32:17,563 - synapse.crypto.keyring - 867 - ERROR - ServerKeyFetcher-112- Error getting keys from vac.systemec.nl Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction r = func(cursor, *args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn return self.simple_upsert_txn_native_upsert( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert txn.execute(sql, list(allvalues.values())) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute self._do_execute(self.txn.execute, sql, parameters) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute return func(sql, args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ psycopg2.OperationalError: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func return func(db_conn, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction conn.rollback() File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback self.conn.rollback() File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 66, in rollback raise ConnectionLost() twisted.enterprise.adbapi.ConnectionLost

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 862, in get_keys keys = await self.get_server_verify_keys_v2_direct(server_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 919, in get_server_verify_keys_v2_direct return await self.process_v2_response( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/crypto/keyring.py", line 601, in process_v2_response await make_deferred_yieldable( File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/keys.py", line 170, in store_server_keys_json await self.db_pool.simple_upsert( File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1243, in simple_upsert return await self.runInteraction( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction return await delay_cancellation(_runInteraction()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction result = await self.runWithConnection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection return await make_deferred_yieldable( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(*args, *kw) ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection result = func(conn, args, kw) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1021, in inner_func self.engine.attempt_to_set_autocommit(conn, False) File "/usr/local/lib/python3.11/site-packages/synapse/storage/engines/postgres.py", line 220, in attempt_to_set_autocommit return conn.set_session(autocommit=autocommit) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ psycopg2.InterfaceError: connection already closed 2023-09-17 15:32:17,582 - synapse.federation.transport.server._base - 312 - WARNING - PUT-569862- authenticate_request failed: 401: Failed to find any key to satisfy: _FetchKeyRequest(server_name='vac.systemec.nl', minimum_valid_until_ts=1694957536141, key_ids=['ed25519:a_MLcI']) 2023-09-17 15:32:17,585 - synapse.http.server - 118 - INFO - PUT-569862- <XForwardedForRequest at 0x7fdb28f94650 method='PUT' uri='/_matrix/federation/v1/send/1680639381906' clientproto='HTTP/1.1' site='8008'> SynapseError: 401 - Failed to find any key to satisfy: _FetchKeyRequest(server_name='vac.systemec.nl', minimum_valid_until_ts=1694957536141, key_ids=['ed25519:a_MLcI']) 2023-09-17 15:32:17,596 - synapse.access.http.8008 - 465 - INFO - PUT-569862- 89.20.83.36 - 8008 - {None} Processed request: 1.448sec/0.008sec (0.007sec, 0.001sec) (0.000sec/0.000sec/0) 187B 401 "PUT /_matrix/federation/v1/send/1680639381906 HTTP/1.1" "Synapse/1.80.0" [0 dbevts] 2023-09-17 15:32:20,338 - synapse.storage.txn - 765 - WARNING - prune_old_user_ips-472440- [TXN OPERROR] {_prune_old_user_ips-2f58d1} server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. 0/5 2023-09-17 15:32:20,341 - twisted - 275 - CRITICAL - prune_old_user_ips-472440- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection result = func(conn, *args, *kw) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func return func(db_conn, args, **kwargs) File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 776, in new_transaction conn.rollback() File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 207, in rollback self.conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:20,345 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback self._connection.rollback() psycopg2.InterfaceError: connection already closed

2023-09-17 15:32:20,348 - twisted - 275 - CRITICAL - sentinel- Rollback failed Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, *kw) File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext return func(args, **kw) --- --- File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection conn.rollback() File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback self._pool.disconnect(self._connection) File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect raise Exception("wrong connection for thread") builtins.Exception: wrong connection for thread

clokep commented 1 year ago

builtins.Exception: wrong connection for thread

This is a very unrelated error, see #16333. If that doesn't fix your issue please file a separate issue.

clokep commented 1 year ago

It looks like Synapse 1.83.0 did the following operations:

I'm suspicious that bumping the schema version twice isn't safe as we have CI checks which compare if develop is +/- of the PR version.

@Anthial The SQL in https://github.com/matrix-org/synapse/issues/16335#issuecomment-1723414709 would be interesting to see still.

Anthial commented 1 year ago

Yes! I'm on it! Just trying to figure out the best way of accessing the SQLite database! I figure I might just back it up once more, and use a DB browser to find the required information! (There seems to be a lack of documentation on how to access the SQLite database when utilising Docker)

Anthial commented 1 year ago

Something like?

Version | file
73 | 73/01event_failed_pull_attempts.sql
-- | --
73 | 73/02add_pusher_enabled.sql
73 | 73/02room_id_indexes_for_purging.sql
73 | 73/03pusher_device_id.sql
73 | 73/03users_approved_column.sql
73 | 73/04partial_join_details.sql
73 | 73/04pending_device_list_updates.sql
73 | 73/05old_push_actions.sql.sqlite
73 | 73/06thread_notifications_thread_id_idx.sql
73 | 73/08thread_receipts_non_null.sql.sqlite
73 | 73/09partial_joined_via_destination.sql
73 | 73/09threads_table.sql
73 | 73/10_update_sqlite_fts4_tokenizer.py
73 | 73/10login_tokens.sql
73 | 73/12refactor_device_list_outbound_pokes.sql
73 | 73/13add_device_lists_index.sql
73 | 73/20_un_partial_stated_room_stream.sql
73 | 73/22_rebuild_user_dir_stats.sql
73 | 73/22_un_partial_stated_event_stream.sql
73 | 73/23_fix_thread_index.sql
73 | 73/24_events_jump_to_date_index.sql
73 | 73/25drop_presence.sql
74 | 74/01_user_directory_stale_remote_users.sql
74 | 74/01membership_tables_event_stream_ordering.sql
74 | 74/02_set_device_id_for_pushers_bg_update.sql
74 | 74/02membership_tables_event_stream_ordering_triggers.py
76 | 76/01_add_profiles_full_user_id_column.sql
76 | 76/02_add_user_filters_full_user_id_column.sql
76 | 76/03_per_user_experimental_features.sql
76 | 76/04_add_room_forgetter.sql
77 | 77/03bg_populate_full_user_id_profiles.sql
77 | 77/04bg_populate_full_user_id_user_filters.sql
77 | 77/05thread_notifications_backfill.sql
77 | 77/06thread_notifications_not_null.sql.sqlite
77 | 77/14bg_indices_event_stream_ordering.sql
78 | 78/01_validate_and_update_profiles.py
78 | 78/02_validate_and_update_user_filters.py
78 | 78/03_remove_unused_indexes_user_filters.py
78 | 78/04_add_full_user_id_index_user_filters.py
78 | 78/03event_extremities_constraints.py
79 | 79/03_read_write_locks_triggers.sql.sqlite
79 | 79/04_mitigate_stream_ordering_update_race.py
79 | 79/05_read_write_locks_triggers.sql.sqlite
80 | 80/01_users_alter_locked.sql
80 | 80/02_scheduled_tasks.sql
clokep commented 1 year ago

Thanks! It looks like you're missing 3 deltas:

Anthial commented 1 year ago

Very strange! I wonder how that could have happened!

Anthial commented 1 year ago

Is there anyway to apply these deltas to the database or is it permanently broken?

clokep commented 1 year ago

Is there anyway to apply these deltas to the database or is it permanently broken?

You could try to apply them by hand, in order. (And hope no invalid data has occurred in the meantime.) I'd love to know how you database ended up in this state though.

clokep commented 1 year ago

I'm suspicious that bumping the schema version twice isn't safe as we have CI checks which compare if develop is +/- of the PR version.

11559 seems to describe the concern I have.

Anthial commented 1 year ago

Alright! I'm going to try to run each of these commands (from each of these deltas, including one more, since I also seem to be missing 03_room_membership_index.sql)! Wish me luck! 😁

I really hope this is just, like, a one or two-time occurrence and that most folks manage to avoid this kind of weirdness!

Anthial commented 1 year ago

SQL manually is always a little bit scary BUT IT WORKS! Thank you so much @clokep! You are the best! Feel free to ask if you want to see any kind of logs, and I'll try to get them for you!

clokep commented 1 year ago

Glad that it worked! I'll poke the team and see if anyone has ideas (there's 1 or 2 things I want to try also.)

erikjohnston commented 1 year ago

We should be correctly handling jumps in schema versions:

https://github.com/matrix-org/synapse/blob/2763c49eca483dbb848b70b951891afd57016f17/synapse/storage/prepare_database.py#L424

There was multiple upgrades and downgrades of schema version in v1.80.0..v1.83.0. @Anthial I assume you weren't running against the develop branch (or prereleases?) at any point?

Otherwise, the schema versions seem sane between v1.70.0 and v1.90.0:

This shows the schema version at each release, and the *new* files for each version. ``` v1.70.0: New Schema Version: 73 synapse/storage/schema/main/delta/73/06thread_notifications_thread_id_idx.sql | 23 +++++++++++++++++++++++ synapse/storage/schema/main/delta/73/08thread_receipts_non_null.sql.postgres | 23 +++++++++++++++++++++++ synapse/storage/schema/main/delta/73/08thread_receipts_non_null.sql.sqlite | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/73/09partial_joined_via_destination.sql | 18 ++++++++++++++++++ synapse/storage/schema/main/delta/73/09threads_table.sql | 30 ++++++++++++++++++++++++++++++ 5 files changed, 170 insertions(+) v1.71.0: New Schema Version: 73 synapse/storage/schema/main/delta/73/10_update_sqlite_fts4_tokenizer.py | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/73/10login_tokens.sql | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+) v1.72.0: New Schema Version: 73 synapse/storage/schema/main/delta/73/11event_search_room_id_n_distinct.sql.postgres | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) v1.73.0: New Schema Version: 73 synapse/storage/schema/main/delta/73/12refactor_device_list_outbound_pokes.sql | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/73/13add_device_lists_index.sql | 20 ++++++++++++++++++++ 2 files changed, 73 insertions(+) v1.74.0: New Schema Version: 73 synapse/storage/schema/main/delta/73/20_un_partial_stated_room_stream.sql | 32 ++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/73/21_un_partial_stated_room_stream_seq.sql.postgres | 20 ++++++++++++++++++++ synapse/storage/schema/main/delta/73/22_rebuild_user_dir_stats.sql | 29 +++++++++++++++++++++++++++++ 3 files changed, 81 insertions(+) v1.75.0: New Schema Version: 73 synapse/storage/schema/main/delta/73/22_un_partial_stated_event_stream.sql | 34 ++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/73/23_fix_thread_index.sql | 33 +++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/73/23_un_partial_stated_room_stream_seq.sql.postgres | 20 ++++++++++++++++++++ 3 files changed, 87 insertions(+) v1.76.0: New Schema Version: 73 synapse/storage/schema/main/delta/73/24_events_jump_to_date_index.sql | 17 +++++++++++++++++ synapse/storage/schema/main/delta/73/25drop_presence.sql | 17 +++++++++++++++++ 2 files changed, 34 insertions(+) v1.77.0: New Schema Version: 74 v1.78.0: New Schema Version: 74 v1.79.0: New Schema Version: 74 v1.80.0: New Schema Version: 74 synapse/storage/schema/main/delta/74/01_user_directory_stale_remote_users.sql | 39 +++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/74/90COMMENTS_destinations.sql.postgres | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) v1.81.0: New Schema Version: 74 synapse/storage/schema/main/delta/74/02_set_device_id_for_pushers_bg_update.sql | 19 +++++++++++++++++++ synapse/storage/schema/main/delta/74/03_room_membership_index.sql | 19 +++++++++++++++++++ 2 files changed, 38 insertions(+) v1.82.0: New Schema Version: 74 synapse/storage/schema/main/delta/74/04_delete_e2e_backup_keys_for_deactivated_users.sql | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) v1.83.0: New Schema Version: 76 synapse/storage/schema/main/delta/74/03_membership_tables_event_stream_ordering.sql.postgres | 29 +++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/74/03_membership_tables_event_stream_ordering.sql.sqlite | 23 +++++++++++++++++++++++ synapse/storage/schema/main/delta/74/04_membership_tables_event_stream_ordering_triggers.py | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/74/05_events_txn_id_device_id.sql | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/76/01_add_profiles_full_user_id_column.sql | 20 ++++++++++++++++++++ synapse/storage/schema/main/delta/76/02_add_user_filters_full_user_id_column.sql | 20 ++++++++++++++++++++ synapse/storage/schema/main/delta/76/03_per_user_experimental_features.sql | 27 +++++++++++++++++++++++++++ 7 files changed, 251 insertions(+) v1.84.0: New Schema Version: 76 synapse/storage/schema/main/delta/76/04_add_room_forgetter.sql | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) v1.85.0: New Schema Version: 77 synapse/storage/schema/main/delta/77/01_add_profiles_not_valid_check.sql.postgres | 16 +++++++++++++++ synapse/storage/schema/main/delta/77/02_add_user_filters_not_valid_check.sql.postgres | 16 +++++++++++++++ synapse/storage/schema/main/delta/77/03bg_populate_full_user_id_profiles.sql | 16 +++++++++++++++ synapse/storage/schema/main/delta/77/04bg_populate_full_user_id_user_filters.sql | 16 +++++++++++++++ synapse/storage/schema/main/delta/77/05thread_notifications_backfill.sql | 28 ++++++++++++++++++++++++++ synapse/storage/schema/main/delta/77/06thread_notifications_not_null.sql.sqlite | 102 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/77/06thread_notifications_not_null_event_push_actions.sql.postgres | 27 +++++++++++++++++++++++++ synapse/storage/schema/main/delta/77/06thread_notifications_not_null_event_push_actions_staging.sql.postgres | 27 +++++++++++++++++++++++++ synapse/storage/schema/main/delta/77/06thread_notifications_not_null_event_push_summary.sql.postgres | 29 +++++++++++++++++++++++++++ synapse/storage/schema/main/delta/77/14bg_indices_event_stream_ordering.sql | 20 +++++++++++++++++++ 10 files changed, 297 insertions(+) v1.86.0: New Schema Version: 78 synapse/storage/schema/main/delta/78/01_validate_and_update_profiles.py | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/78/02_validate_and_update_user_filters.py | 95 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 187 insertions(+) v1.87.0: New Schema Version: 78 synapse/storage/schema/main/delta/78/03_remove_unused_indexes_user_filters.py | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/78/04_add_full_user_id_index_user_filters.py | 25 +++++++++++++++++++++++++ 2 files changed, 90 insertions(+) v1.88.0: New Schema Version: 78 synapse/storage/schema/main/delta/78/03event_extremities_constraints.py | 57 ++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/78/04_read_write_locks_triggers.sql.postgres | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/78/04_read_write_locks_triggers.sql.sqlite | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 328 insertions(+) v1.89.0: New Schema Version: 79 synapse/storage/schema/main/delta/79/04_mitigate_stream_ordering_update_race.py | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/79/05_read_write_locks_triggers.sql.postgres | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/79/05_read_write_locks_triggers.sql.sqlite | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 204 insertions(+) v1.90.0: New Schema Version: 80 ``` Command: `for i in $( seq 70 90 ); do echo -n "v1.$i.0: New Schema Version: "; git checkout -q "v1.$i.0"; python -c "from __init__ import SCHEMA_VERSION; print(SCHEMA_VERSION)"; git diff --stat --diff-filter=A v1.$(($i-1)).0..v1.$i.0 .; echo ""; git checkout -q ~/git/synapse; done`

The changes to the schemas (only showing addition/deletions) in the v1.80 to v1.83 range are:

``` v1.80.0rc2: Schema: 75. Commit: e6af49fbea Reintroduce membership tables event stream ordering (#15128) — 2023-03-24 11:44:01 +0000 synapse/storage/schema/main/delta/74/01membership_tables_event_stream_ordering.sql | 20 ++++++++++++++++++++ synapse/storage/schema/main/delta/74/02membership_tables_event_stream_ordering_triggers.py | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+) v1.80.0rc2: Schema: 75. Commit: 5b70f240cf Make cleaning up pushers depend on the device_id instead of the token_id (#15280) — 2023-03-24 11:09:39 -0400 synapse/storage/schema/main/delta/74/02_set_device_id_for_pushers_bg_update.sql | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) v1.80.0: Schema: 74. Commit: 5350b5d04d Revert "Reintroduce membership tables event stream ordering (#15128)" (#15347) — 2023-03-29 13:24:28 +0100 synapse/storage/schema/main/delta/74/01membership_tables_event_stream_ordering.sql | 20 -------------------- synapse/storage/schema/main/delta/74/02membership_tables_event_stream_ordering_triggers.py | 79 ------------------------------------------------------------------------------- 2 files changed, 99 deletions(-) v1.80.0: Schema: 74. Commit: 2a234b788e Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15350) — 2023-03-30 15:11:31 -0400 synapse/storage/schema/main/delta/74/02thread_notifications_backfill.sql | 28 ++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/74/03thread_notifications_not_null.sql.postgres | 23 +++++++++++++++++++++++ synapse/storage/schema/main/delta/74/03thread_notifications_not_null.sql.sqlite | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 150 insertions(+) v1.80.0: Schema: 74. Commit: 72d2ceaa9a Revert "Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15350)" — 2023-03-31 12:10:10 +0100 synapse/storage/schema/main/delta/74/02thread_notifications_backfill.sql | 28 ---------------------------- synapse/storage/schema/main/delta/74/03thread_notifications_not_null.sql.postgres | 23 ----------------------- synapse/storage/schema/main/delta/74/03thread_notifications_not_null.sql.sqlite | 99 --------------------------------------------------------------------------------------------------- 3 files changed, 150 deletions(-) v1.80.0: Schema: 74. Commit: 79d2e2e79c Speed up membership queries for users with forgotten rooms (#15385) — 2023-04-04 14:11:34 +0100 synapse/storage/schema/main/delta/74/03_room_membership_index.sql | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) v1.81.0rc1: Schema: 74. Commit: 6b23d74ad1 Delete server-side backup keys when deactivating an account. (#15181) — 2023-04-04 20:16:08 +0000 synapse/storage/schema/main/delta/74/04_delete_e2e_backup_keys_for_deactivated_users.sql | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) v1.82.0rc1: Schema: 74. Commit: 8b3a502996 Experimental support for MSC3970: per-device transaction IDs (#15318) — 2023-04-25 09:37:09 +0100 synapse/storage/schema/main/delta/74/05_events_txn_id_device_id.sql | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) v1.82.0rc1: Schema: 75. Commit: c55293c230 Re re introduce membership tables event stream ordering (#15356) — 2023-04-25 09:44:29 +0100 synapse/storage/schema/main/delta/74/03_membership_tables_event_stream_ordering.sql.postgres | 29 +++++++++++++++++++++++++++++ synapse/storage/schema/main/delta/74/03_membership_tables_event_stream_ordering.sql.sqlite | 23 +++++++++++++++++++++++ synapse/storage/schema/main/delta/74/04_membership_tables_event_stream_ordering_triggers.py | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 131 insertions(+) v1.82.0: Schema: 76. Commit: 301b4156d5 Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458) — 2023-04-26 16:03:26 -0700 synapse/storage/schema/main/delta/76/01_add_profiles_full_user_id_column.sql | 20 ++++++++++++++++++++ synapse/storage/schema/main/delta/76/02_add_user_filters_full_user_id_column.sql | 20 ++++++++++++++++++++ 2 files changed, 40 insertions(+) v1.82.0: Schema: 76. Commit: 89f6fb0d5a Add an admin API endpoint to support per-user feature flags (#15344) — 2023-04-28 11:33:45 -0700 synapse/storage/schema/main/delta/76/03_per_user_experimental_features.sql | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) ``` Command: `for c in e6af49fbea 5b70f240cf 5350b5d04d 2a234b788e 72d2ceaa9a 79d2e2e79c 6b23d74ad1 8b3a502996 c55293c230 301b4156d5 89f6fb0d5a; do git checkout -q $c; VERSION=$(git describe --tags --abbrev=0); SCHEMA=$( python -c "from __init__ import SCHEMA_VERSION; print(SCHEMA_VERSION)" ); date=$(git show -s --format=%ci); oneline=$(git show -s --oneline); echo "$VERSION: Schema: $SCHEMA. Commit: $oneline — $date"; git diff --stat --diff-filter=AD HEAD^ .; echo ""; git checkout -q ~/git/synapse; done`
Anthial commented 1 year ago

As far as I am aware, I did not run the develop branch or any of the prereleases!

erikjohnston commented 11 months ago

I can't see how this can happen without running the develop branch somehow, though seems like it did. I'm wondering if there was some weird quirk in the setup here? Given no one else is complaining about similar problems, and I don't see a way of progressing this issue, I'm going to close it.

If anyone else sees this please shout!