Proper way to decommission homeserver from federation to stop inbound traffic

[Valodim]

It would be useful to have an actual way to decommission servers. In theory, servers can leave all rooms and be done with it, but that is non-trivial and in practice servers will get federation requests for basically forever.

A possible measure would be a specific value in the SRV record, or a HTTP response that tells other servers that this server no longer participates in the federation and shouldn't be pinged again.

(kind of related to #3286, but for federation traffic instead of client traffic)

[hrefhref]

Maybe returning a HTTP 410 ("Gone") at all /_matrix/ endpoints?

[bmisiak]

@hrefhref I feel like a DNS entry would be preferable as it would help avoid unnecessary requests to the server.

[Scrumplex]

I would suggest something like

example.com.    1    IN    TXT    "v=matrix;gone"

where example.com is the actual domain that was used for the homeserver.

[Valodim]

There is already a SRV record that needs to be resolved, might as well just define the semantics of some domain to mean it's decommissioned. The .invalid TLD seems well suited for the job. This boils down the actual logic in the HS to "if SRV resolves to *.invalid, don't bother querying but maybe recheck the record at a later point".

Bonus points if e.g. joining a room or sending a message in a 1on1 to a user on such a server gives a specialized error.

[Scrumplex]

That is a pretty good idea, but we also need to think about when it would be time to forget a homeserver. I would suggest "forgetting" a server 30 days after the SRV record points to something.invalid.

[ghost]

Not a fan of requiring a tombstone to ward off evil spirits.

It would be great if we could come up with a positive way to know that a matrix server is up and running. One of the easiest ways would be to require a valid SRV for federation, but there has to be a better way than that.

[ghost]

@ara4n commented:

<@freenode_andi-:matrix.org> So I intend to stop running my matrix homeserver for farious reasons. Is there a way to tell other servers that mine is gone and they do not have to retry forever?

if the farious reasons relate to history retention, be aware that msc1763 is trying to fix this

[echto]

What if a server has left one federation and joined another? How does a change to the DNS record help that?

[Mikaela]

Feneas.org homeserver will shutdown early UTC on 2022-03-01 due to dissolving the association and I think it has been participating the federation widely so having a proper method for decommissioning could reduce the amount of errors. There is a plan to part all users from all rooms first, but that will not stop clients from attempting to join using feneas.org aliases or matrix/matrix.to uris from containing via=feneas.orgs potentially resulting to ambiguous error messages instead of a clear prompt on the server being shut down.

Previously privacytools.io homeserver shut down (however they lost access to the domain) and before that Disroot.org shut down their Matrix server.

[Mikaela]

another concern: what prevents feneas.org users from getting state reset into rooms after they have left and the homeserver is down? my account was apparently joined some Element room regardless of only hanging in committee and one DM I had to backup for ages and I have seen privacytools.io users state resetting several smaller rooms too

I apologize for double-posting, I just happened to remember another scenario where a proper decommissioning policy would be helpful.

[bkil]

Is it true that a room lives within the home server that it was created on and that any such room will fail after switching off that home server? So is it true that a tombstone redirect will need to be placed to another HS before switching it off?

[clokep]

Is it true that a room lives within the home server that it was created on and that any such room will fail after switching off that home server? So is it true that a tombstone redirect will need to be placed to another HS before switching it off?

No, rooms live on all homeservers which are joined to it. If a homeserver is being decommissioned you'll want to deactivate all the accounts first (so they leave all the rooms). Before leaving those rooms you might want to ensure they have other people who have admin powers.

[bkil]

I tried to save a permalink to a message in such a room, but the ID seems to contain the original home server as well. That sounds to be an issue.

[neilalexander]

I tried to save a permalink to a message in such a room, but the ID seems to contain the original home server as well. That sounds to be an issue.

The room ID typically contains the name of the server that the room was created on, but any server that is participating in the room will be able to resolve this name regardless of whether that server is still alive or not.

[bkil]

Does that mean, that hypothetically (or due to a bug), if every member left the room, they would not be able to access previous room history after they came back? Does Synapse do such garbage collection by default?

[bkil]

It would be nice to a have a room migration facility to cope with this. And I recall that media storage is also tied to the home server of a user. So am I right in that media (and attachment?) posted by such users will also vanish? Again it would be helpful to have a solution for transferring ownership of these.

[Valodim]

This is not a place to ask general questions about how matrix works. Please stay on topic.

[bkil]

We want to ensure that we made every precaution that would be feasible before the Feneas switch-off. My above question was still basically: do we need to tombstone all rooms involved? Do we need to bother with mirroring content?

[aaronraimist]

do we need to tombstone all rooms involved?

No. As already stated, room IDs don't mean anything. Feneas.org going away doesn't change anything about rooms that have room IDs ending in feneas.org. Room IDs only have a domain on the end to ensure they are unique.

Does that mean, that hypothetically (or due to a bug), if every member left the room, they would not be able to access previous room history after they came back?

If every member leaves a room, it becomes unjoinable.

Does Synapse do such garbage collection by default?

No.

So am I right in that media (and attachment?) posted by such users will also vanish?

It will not vanish from servers which have already received that media. No additional servers will be able to fetch it from Feneas.org though. Theoretically servers could fetch the media from other servers in the room but I don't believe this is done right now.

If you have further questions about how Matrix works I would suggest you ask on Matrix, not in github issues.

[Mikaela]

I am not sure if this is the place, but I think decomissioning a homeserver should also include some method of revocating m.room.power_levels events in rooms so in case the domain returns with a new owner who setups a homeserver, they won't instantly get power in all rooms just by making accounts with those names.

I just also expressed this concern in https://github.com/matrix-org/matrix-spec-proposals/pull/3510/files#r848134997 as it's is one of the issues that proposal hopes to resolve while introducing other concerns.

However I wonder if this issue really belongs to matrix-org/synapse instead of matrix-org/matrix-spec(-proposals(?)) as Synapse isn't the only homeserver implementation nowadays and having a proper decomissioning protocol is desirable regardless of which implementation is being used.

[derzahla]

In the 3 years since this issue was opened, has anyone from matrix-org provided guidance on how to exorcise the federation hooks from Synapse? I manage a homeserver that has been operating standalone for many months now but it still spanguis the logs trying to reach out the dozens of external homeservers.

How can I make this stop?

I have already tried the following:

• removed the federation listener from homeserver.yml
• set the federation whitelist to " "
• purged all old rooms that were left over from federation
• manually cleared all rows in the "destinations" and "destination_room" database tables

It would be great to get an answer to https://github.com/matrix-org/synapse/issues/4979 and many similar requests and be provided documentation on what exactly needs to be done to STOP all federation activity and remove all record of previously federated servers?

[richvdh]

@derzahla if your server is making outbound calls, that means it considers that users on your server still share rooms with users on other servers. #4979 has an answer and is closed accordingly. This issue is about inbound traffic, so your question is off-topic here.

[winterqt]

However I wonder if this issue really belongs to matrix-org/synapse instead of matrix-org/matrix-spec(-proposals(?)) as Synapse isn't the only homeserver implementation nowadays and having a proper decomissioning protocol is desirable regardless of which implementation is being used.

I also think this (as well as this discussion) would be better off as an MSC -- would obviously be much more beneficial than a Synapse-specific implementation.

[emilweth]

Any update on this? I have a nginx server that has been responding 410 gone to hundreds of matrix servers, making thousands of requests per day, for more than a year, and this is kinda annoying.

What's the point of writing a protocol based on HTTP if it cannot understand basic HTTP codes?

[thedewi]

I ran a home server briefly, and decommissioned it 17 months ago, and I still get hits from synapse every minute or two. This can't be helpful to anyone. Any way to stop it? The SRV record is long gone.