matrix.org had the wrong value for a signing key

[richvdh]

there are a bunch of events in #offtopic (and probably elsewhere) which are signed by home.rdash.in:ed25519:a_mEAC. Unfortunately matrix.org doesn't have a copy of this key, so synapse tries to get the event direct from the server - which is unresponsive.

Each attempt to fetch the key takes 5 minutes to time out (presumably because it makes multiple attempts), and each request seems to end up getting queued up - so we end up with pauses of hours in the middle of handling a given request.

So then what happens is matrix.org and other servers end up timing out federation send requests, which then breaks everyone's federation

[richvdh]

Unfortunately matrix.org doesn't have a copy of this key

Actually it's even stranger than that. matrix.org has an entry for the key in its database, but the key in the database is actually home.rdash.in:ed25519:a_nsPM:

matrix=# select server_name, key_id, from_server, ts_added_ms, ts_valid_until_ms, encode(key_json, 'escape')  as key_json from server_keys_json where server_name='home.rdash.in';

  server_name  |     key_id     |  from_server  |  ts_added_ms  | ts_valid_until_ms |                                                                                                                                                                                                                                                          key_json                                                                                                                                                                                                                                                          
---------------+----------------+---------------+---------------+-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 home.rdash.in | ed25519:a_dQKK | home.rdash.in | 1450887042978 |     1448830622801 | {"old_verify_keys":{},"server_name":"home.rdash.in","signatures":{"home.rdash.in":{"ed25519:a_dQKK":"VKcrP9g0zdlI++rC3vGUbzaQAXiJGe7WEaTAjYg6IoXFSDjZk0sMxI7OLq6vNrcQo9MntzQr8iI0+SfU7k3cBw"},"matrix.org":{"ed25519:auto":"LSapkJoDsxKyLqw7ZzUJ0NexqzjefSlgpZZHtJn16Gy/a9s6jpxSZ6k0/4Xi/i0A1Pp9TGyK8+0oBBJj1R+rAw"}},"tls_fingerprints":[{"sha256":"Kel/eFHTLYc0vgRZ1tUjmoBa5UCPlbq1Nnklql837CQ"}],"valid_until_ts":1448830622801,"verify_keys":{"ed25519:a_dQKK":{"key":"EcBFBeGsi0Bt7fdc9v9ikJcoICE8pIBF59kZ3Z4v/yY"}}}
 home.rdash.in | ed25519:a_mEAC | home.rdash.in | 1511539204620 |     1511625604574 | {"old_verify_keys":{},"server_name":"home.rdash.in","signatures":{"home.rdash.in":{"ed25519:a_nsPM":"HI2dCJv86JTnftlniEWl9flllKIUHDH+8re+ayvlJ0q6AYSZJyZWWTRGKTdwY5C60s197yZxity6s4XCbDsPDg"},"matrix.org":{"ed25519:auto":"Dx3pXavtLjNbonTxnTyEX2W+x1QaZxbQdnAVDBYzuxE+PloxjXK/DLuxgZh/4fUDBQcY31Dun4tp+cQT9tM8AA"}},"tls_fingerprints":[{"sha256":"/GB+YLXLEGUgh/XKMJ7w7rBJyoIxUfp2Rjv6OHl/jk0"}],"valid_until_ts":1511625604574,"verify_keys":{"ed25519:a_nsPM":{"key":"i90e0gGqbPAv1G7QBvW+MhQT4PgapcL0jrfTsAHtUvY"}}}
 home.rdash.in | ed25519:a_nsPM | home.rdash.in | 1511539204620 |     1511625604574 | {"old_verify_keys":{},"server_name":"home.rdash.in","signatures":{"home.rdash.in":{"ed25519:a_nsPM":"HI2dCJv86JTnftlniEWl9flllKIUHDH+8re+ayvlJ0q6AYSZJyZWWTRGKTdwY5C60s197yZxity6s4XCbDsPDg"},"matrix.org":{"ed25519:auto":"Dx3pXavtLjNbonTxnTyEX2W+x1QaZxbQdnAVDBYzuxE+PloxjXK/DLuxgZh/4fUDBQcY31Dun4tp+cQT9tM8AA"}},"tls_fingerprints":[{"sha256":"/GB+YLXLEGUgh/XKMJ7w7rBJyoIxUfp2Rjv6OHl/jk0"}],"valid_until_ts":1511625604574,"verify_keys":{"ed25519:a_nsPM":{"key":"i90e0gGqbPAv1G7QBvW+MhQT4PgapcL0jrfTsAHtUvY"}}}

[richvdh]

I've updated the database; let's see if that works around the problem.

[richvdh]

the specific question here is "why did matrix.org have a copy of the wrong key". The follow-on question of "why did everything get queued up behind it" is #3825.

[richvdh]

given that this has never been seen before or since, I don't think there's much point in keeping this open.