Closed richvdh closed 2 years ago
Unfortunately matrix.org doesn't have a copy of this key
Actually it's even stranger than that. matrix.org has an entry for the key in its database, but the key in the database is actually home.rdash.in:ed25519:a_nsPM
:
matrix=# select server_name, key_id, from_server, ts_added_ms, ts_valid_until_ms, encode(key_json, 'escape') as key_json from server_keys_json where server_name='home.rdash.in';
server_name | key_id | from_server | ts_added_ms | ts_valid_until_ms | key_json
---------------+----------------+---------------+---------------+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
home.rdash.in | ed25519:a_dQKK | home.rdash.in | 1450887042978 | 1448830622801 | {"old_verify_keys":{},"server_name":"home.rdash.in","signatures":{"home.rdash.in":{"ed25519:a_dQKK":"VKcrP9g0zdlI++rC3vGUbzaQAXiJGe7WEaTAjYg6IoXFSDjZk0sMxI7OLq6vNrcQo9MntzQr8iI0+SfU7k3cBw"},"matrix.org":{"ed25519:auto":"LSapkJoDsxKyLqw7ZzUJ0NexqzjefSlgpZZHtJn16Gy/a9s6jpxSZ6k0/4Xi/i0A1Pp9TGyK8+0oBBJj1R+rAw"}},"tls_fingerprints":[{"sha256":"Kel/eFHTLYc0vgRZ1tUjmoBa5UCPlbq1Nnklql837CQ"}],"valid_until_ts":1448830622801,"verify_keys":{"ed25519:a_dQKK":{"key":"EcBFBeGsi0Bt7fdc9v9ikJcoICE8pIBF59kZ3Z4v/yY"}}}
home.rdash.in | ed25519:a_mEAC | home.rdash.in | 1511539204620 | 1511625604574 | {"old_verify_keys":{},"server_name":"home.rdash.in","signatures":{"home.rdash.in":{"ed25519:a_nsPM":"HI2dCJv86JTnftlniEWl9flllKIUHDH+8re+ayvlJ0q6AYSZJyZWWTRGKTdwY5C60s197yZxity6s4XCbDsPDg"},"matrix.org":{"ed25519:auto":"Dx3pXavtLjNbonTxnTyEX2W+x1QaZxbQdnAVDBYzuxE+PloxjXK/DLuxgZh/4fUDBQcY31Dun4tp+cQT9tM8AA"}},"tls_fingerprints":[{"sha256":"/GB+YLXLEGUgh/XKMJ7w7rBJyoIxUfp2Rjv6OHl/jk0"}],"valid_until_ts":1511625604574,"verify_keys":{"ed25519:a_nsPM":{"key":"i90e0gGqbPAv1G7QBvW+MhQT4PgapcL0jrfTsAHtUvY"}}}
home.rdash.in | ed25519:a_nsPM | home.rdash.in | 1511539204620 | 1511625604574 | {"old_verify_keys":{},"server_name":"home.rdash.in","signatures":{"home.rdash.in":{"ed25519:a_nsPM":"HI2dCJv86JTnftlniEWl9flllKIUHDH+8re+ayvlJ0q6AYSZJyZWWTRGKTdwY5C60s197yZxity6s4XCbDsPDg"},"matrix.org":{"ed25519:auto":"Dx3pXavtLjNbonTxnTyEX2W+x1QaZxbQdnAVDBYzuxE+PloxjXK/DLuxgZh/4fUDBQcY31Dun4tp+cQT9tM8AA"}},"tls_fingerprints":[{"sha256":"/GB+YLXLEGUgh/XKMJ7w7rBJyoIxUfp2Rjv6OHl/jk0"}],"valid_until_ts":1511625604574,"verify_keys":{"ed25519:a_nsPM":{"key":"i90e0gGqbPAv1G7QBvW+MhQT4PgapcL0jrfTsAHtUvY"}}}
I've updated the database; let's see if that works around the problem.
the specific question here is "why did matrix.org have a copy of the wrong key". The follow-on question of "why did everything get queued up behind it" is #3825.
given that this has never been seen before or since, I don't think there's much point in keeping this open.
there are a bunch of events in #offtopic (and probably elsewhere) which are signed by
home.rdash.in:ed25519:a_mEAC
. Unfortunately matrix.org doesn't have a copy of this key, so synapse tries to get the event direct from the server - which is unresponsive.Each attempt to fetch the key takes 5 minutes to time out (presumably because it makes multiple attempts), and each request seems to end up getting queued up - so we end up with pauses of hours in the middle of handling a given request.
So then what happens is matrix.org and other servers end up timing out federation send requests, which then breaks everyone's federation