element-hq / element-meta

Shared/meta documentation and project artefacts for Element clients
74 stars 12 forks source link

Failed to upgrade room version #2530

Open ara4n opened 2 months ago

ara4n commented 2 months ago

Steps to reproduce

  1. Went to Element Web/Desktop (~15k members iirc)
  2. /upgraderoom 11
  3. Much spinning, and then a "fetch failed to fetch" error msg.
  4. New room got created, seemingly with the right state.
  5. Had to manually send a tombstone event in the old room to point to the new room.

Outcome

What did you expect?

Reliable room upgrades

What happened instead?

Screenshot 2024-07-29 at 15 24 18

Operating system

No response

Application version

Element Nightly version: 2024072901 Crypto version: Rust SDK 0.7.1 (c8c9d15), Vodozemac 0.6.0

How did you install the app?

No response

Homeserver

matrix.org

Will you send logs?

Yes

t3chguy commented 2 months ago

This sounds like a server issue, we just call the /upgrade endpoint and the HS is responsible for sending the tombstone.

t3chguy commented 2 months ago

Would need Synapse logs to investigate. Looks like the request timed out after 10 seconds presumably due to no keepalive from Synapse

ara4n commented 2 months ago

10 seconds is a very short timeout, especially for important operations, especially if they could take more than 10s to execute...

t3chguy commented 2 months ago

Agreed, but that timeout isn't client side.

2024-07-29T14:23:58.641Z D FetchHttpApi: <-- POST https://matrix-client.matrix.org/_matrix/client/v3/rooms/!YTvKGNlinIzlkMTVRl%3Amatrix.org/upgrade [100067ms TypeError: Failed to fetch]
2024-07-29T14:23:58.642Z E fetch failed: Failed to fetch
ConnectionError: fetch failed: Failed to fetch
    at MatrixHttpApi.requestOtherUrl (vector://vector/webapp/bundles/4973b953927618fdf5b2/init.js:32134:13)
    at async MatrixHttpApi.authedRequest (vector://vector/webapp/bundles/4973b953927618fdf5b2/init.js:31980:24)
    at async upgradeRoom (vector://vector/webapp/bundles/4973b953927618fdf5b2/init.js:151567:9)
    at async vector://vector/webapp/bundles/4973b953927618fdf5b2/init.js:71390:9
    at async runSlashCommand (vector://vector/webapp/bundles/4973b953927618fdf5b2/init.js:123663:9)
    at async SendMessageComposer.sendMessage (vector://vector/webapp/bundles/4973b953927618fdf5b2/init.js:116627:40)

It was timed out from under us, likely on the server side

juliusv commented 1 month ago

As requested by @ara4n on chat, sharing what happened when upgrading two rooms (https://app.element.io/#/room/#prometheus-dev:matrix.org and https://app.element.io/#/room/#prometheus:matrix.org) from v1 to v10 using /updateroom 10:

image

Now we have no Admins at all left in the new room and can no longer admin it.