search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.81k stars 2.13k forks source link

synapse_port_db is completely broken by the new user_directory_search table #2287

Closed ara4n closed 7 years ago

saintger commented 7 years ago

I can confirm that synapse_port_db doesn't currently work (I got the same error as in https://github.com/matrix-org/synapse/issues/2346).

mpiscaer commented 7 years ago

Here fails on the error: FirstError: FirstError[#12, [Failure instance: Traceback: <class 'psycopg2.ProgrammingError'>: column "value" of relation "user_directory_search" does not exist

So it can't find the value column.

saintger commented 7 years ago

Matthew proposed a patch here (untested !): https://matrix.to/#/!cURbafjkfsMDVwdRDQ:matrix.org/$1499810180629967ijWGR:matrix.org

matrix@phoenix:~/synapse/scripts$ git diff
diff --git a/scripts/synapse_port_db b/scripts/synapse_port_db
index 7d158a4..348a59c 100755
--- a/scripts/synapse_port_db
+++ b/scripts/synapse_port_db
@@ -246,6 +246,13 @@ class Porter(object):

         self.progress.add_table(table, postgres_size, table_size)

+        if table == "user_directory_search":
+            # FIXME: actually port it, but for now we can leave it blank
+            # and have the server regenerate it.
+            # you will need to set the values of user_directory_stream_pos
+            # to be ('X', null) to force a regen
+            return
+
         if table == "event_search":
             yield self.handle_search_table(
                 postgres_size, table_size, forward_chunk, backward_chunk
saintger commented 7 years ago

Got this error in the log if I use the previous patch:

synapse.metrics - 162 - INFO - Collecting gc 2
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v17
synapse.storage.prepare_database - 256 - INFO - Applying schema 17/drop_indexes.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 17/server_keys.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 17/user_threepids.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v18
synapse.storage.prepare_database - 256 - INFO - Applying schema 18/server_keys_bigger_ints.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v19
synapse.storage.prepare_database - 256 - INFO - Applying schema 19/event_index.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v20
synapse.storage.prepare_database - 256 - INFO - Applying schema 20/dummy.sql
synapse.storage.prepare_database - 245 - INFO - Running script 20/pushers.py
synapse.storage.v20_pushers - 31 - INFO - Porting pushers table...
synapse.storage.v20_pushers - 76 - INFO - Moved 0 pushers to new table
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v21
synapse.storage.prepare_database - 256 - INFO - Applying schema 21/end_to_end_keys.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 21/receipts.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v22
synapse.storage.prepare_database - 256 - INFO - Applying schema 22/receipts_index.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 22/user_threepids_unique.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v23
synapse.storage.prepare_database - 256 - INFO - Applying schema 23/drop_state_index.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 23/refresh_tokens.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v24
synapse.storage.prepare_database - 256 - INFO - Applying schema 24/stats_reporting.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v25
synapse.storage.prepare_database - 256 - INFO - Applying schema 25/00background_updates.sql
synapse.storage.prepare_database - 245 - INFO - Running script 25/fts.py
synapse.storage.prepare_database - 256 - INFO - Applying schema 25/guest_access.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 25/history_visibility.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 25/tags.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v26
synapse.storage.prepare_database - 256 - INFO - Applying schema 26/account_data.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v27
synapse.storage.prepare_database - 256 - INFO - Applying schema 27/account_data.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 27/forgotten_memberships.sql
synapse.storage.prepare_database - 245 - INFO - Running script 27/ts.py
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v28
synapse.storage.prepare_database - 256 - INFO - Applying schema 28/event_push_actions.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 28/events_room_stream.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 28/public_roms_index.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 28/receipts_user_id_index.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 28/upgrade_times.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 28/users_is_guest.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v29
synapse.storage.prepare_database - 256 - INFO - Applying schema 29/push_actions.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v30
synapse.storage.prepare_database - 256 - INFO - Applying schema 30/alias_creator.sql
synapse.storage.prepare_database - 245 - INFO - Running script 30/as_users.py
synapse.storage.prepare_database - 256 - INFO - Applying schema 30/deleted_pushers.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 30/presence_stream.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 30/public_rooms.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 30/push_rule_stream.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 30/state_stream.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 30/threepid_guest_access_tokens.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v31
synapse.storage.prepare_database - 256 - INFO - Applying schema 31/invites.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 31/local_media_repository_url_cache.sql
synapse.storage.prepare_database - 245 - INFO - Running script 31/pushers.py
synapse.storage.v31_pushers - 31 - INFO - Porting pushers table, delta 31...
synapse.storage.v31_pushers - 75 - INFO - Moved 0 pushers to new table
synapse.storage.prepare_database - 256 - INFO - Applying schema 31/pushers_index.sql
synapse.storage.prepare_database - 245 - INFO - Running script 31/search_update.py
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v32
synapse.storage.prepare_database - 256 - INFO - Applying schema 32/events.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 32/openid.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 32/pusher_throttle.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 32/remove_indices.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 32/reports.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v33
synapse.storage.prepare_database - 256 - INFO - Applying schema 33/access_tokens_device_index.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 33/devices.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 33/devices_for_e2e_keys.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 33/devices_for_e2e_keys_clear_unknown_device.sql
synapse.storage.prepare_database - 245 - INFO - Running script 33/event_fields.py
synapse.storage.prepare_database - 256 - INFO - Applying schema 33/refreshtoken_device.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 33/refreshtoken_device_index.sql
synapse.storage.prepare_database - 245 - INFO - Running script 33/remote_media_ts.py
synapse.storage.prepare_database - 256 - INFO - Applying schema 33/user_ips_index.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v34
synapse.storage.prepare_database - 256 - INFO - Applying schema 34/appservice_stream.sql
synapse.storage.prepare_database - 245 - INFO - Running script 34/cache_stream.py
synapse.storage.prepare_database - 256 - INFO - Applying schema 34/device_inbox.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 34/push_display_name_rename.sql
synapse.storage.prepare_database - 245 - INFO - Running script 34/received_txn_purge.py
synapse.storage.prepare_database - 245 - INFO - Running script 34/sent_txn_purge.py
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v35
synapse.storage.prepare_database - 256 - INFO - Applying schema 35/add_state_index.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 35/contains_url.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 35/device_outbox.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 35/device_stream_id.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 35/event_push_actions_index.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 35/public_room_list_change_stream.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 35/state.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 35/state_dedupe.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 35/stream_order_to_extrem.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v36
synapse.storage.prepare_database - 256 - INFO - Applying schema 36/readd_public_rooms.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v37
synapse.storage.prepare_database - 245 - INFO - Running script 37/remove_auth_idx.py
synapse.storage.prepare_database - 256 - INFO - Applying schema 37/user_threepids.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v38
synapse.storage.prepare_database - 256 - INFO - Applying schema 38/postgres_fts_gist.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v39
synapse.storage.prepare_database - 256 - INFO - Applying schema 39/appservice_room_list.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 39/device_federation_stream_idx.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 39/event_push_index.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 39/federation_out_position.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 39/membership_profile.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v40
synapse.storage.prepare_database - 256 - INFO - Applying schema 40/current_state_idx.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 40/device_inbox.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 40/device_list_streams.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 40/event_push_summary.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 40/pushers.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v41
synapse.storage.prepare_database - 256 - INFO - Applying schema 41/device_list_stream_idx.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 41/device_outbound_index.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 41/event_search_event_id_idx.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 41/ratelimit.sql
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v42
synapse.storage.prepare_database - 256 - INFO - Applying schema 42/current_state_delta.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 42/device_list_last_id.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 42/event_auth_state_only.sql
synapse.storage.prepare_database - 245 - INFO - Running script 42/user_dir.py
synapse.storage.prepare_database - 212 - INFO - Upgrading schema to v43
synapse.storage.prepare_database - 256 - INFO - Applying schema 43/blocked_rooms.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 43/quarantine_media.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 43/url_cache.sql
synapse.storage.prepare_database - 256 - INFO - Applying schema 43/user_share.sql
synapse_port_db - 466 - INFO - Found 110 tables
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 1
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 2
synapse.metrics - 162 - INFO - Collecting gc 0
synapse_port_db - 466 - INFO - Found 110 tables
synapse_port_db - 496 - INFO - Failed to create port table: ERROR:  column « rowid » does not exist
synapse_port_db - 503 - INFO - Failed to create port table: ERROR:  relation « port_from_sqlite3 » already exists
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
synapse.metrics - 162 - INFO - Collecting gc 0
krombel commented 7 years ago

AFAIK this issue got fixed with #2375

richvdh commented 7 years ago

thanks @krombel