Migrate off EMS to self-hosted Synapse

delroth commented 8 months ago

Filing this bug to track actions required for the EMS legacy plans deprecation. We think that the nixos.org Matrix hosting is impacted, but we don't know for sure because we think only @grahamc is receiving notifications related to this account.

Options:

We're not impacted: nothing to do.
We are impacted: we need to decide quickly on one of those two options, because the default is the data gets deleted on Jan 17th 2024.
- We stay on EMS: EMS is doing a special offer upgrading affected customers to their Business plan while keeping feature parity with legacy plans, at a minimum of 20 users. That would increase our hosting costs to $100/month, to pay annually.
- We move to self-host: EMS can provide us with a database dump which we could import. We'd need to run our own Synapse on non-critical-infra.

grahamc commented 8 months ago

We are impacted. I've contacted them to see if they're able to sponsor the instance. This is the email we got:


Dear Graham Christensen,                                                                                   This is the second reminder regarding the communication we have previously sent.                               We are writing to inform you we are discontinuing our consumer hosting products and legacy plans.                               Therefore on Wednesday, January 17th 2024, your current subscription for nixos.ems.host homeserver will come to an end.                                                                                                                                               What happens next?                                                                                                                If you want to keep your service active, you can either:                                                                                                                  1. Upgrade to an Element Business subscription with reduced minimum user commitments (see below)                                                                                                                                               2. Or, if you have a custom domain, you can download a copy of your data and migrate to another Matrix hosting provider (or host your own homeserver)                                                                                                                                               Alternatively, if you are no longer using the service, you can proactively cancel your subscription or do                               nothing and your server will be automatically decommissioned on Wednesday, January 17th 2024.                                                                                                                For existing customers who wish to upgrade to an Element Business subscription, we are exceptionally offering a reduced minimum user requirement of 20 users (usually 50 users) at $5 per user, per month (excluding tax).                               This would have to be paid annually in advance. The plan includes everything covered by your previous                               subscription, plus other features such as single sign-on support. For more information about Element                               Business and our other plans please visit our website.                                                                                                                If you would like to upgrade your hosting plan, please contact our Support team before Wednesday, January 17th 2024                               through our web portal.                                                                                                                To migrate your data to another provider or host your own homeserver, please see our documentation about migrating away from Element.                                                                                   If we do not hear from you by Wednesday, January 17th 2024, we will cancel your subscription for nixos.ems.host,                               along with any additional subscriptions linked to this host (e.g. bridges), shut down your service and                               delete all associated data. You can also cancel your subscription before the termination date via our web portal.                                                                                                                                               We are truly sorry for any inconvenience this causes, unfortunately we can’t support these services anymore.                               You can find alternative Matrix hosting providers on the Matrix website.                                                                                                                If you have any questions on the information above please don’t hesitate to contact our Support team, ems-support@element.io.                                                                                   Thanks,                               The Element Team | Dear Graham Christensen, | This is the second reminder regarding the communication we have previously sent.                               We are writing to inform you we are discontinuing our consumer hosting products and legacy plans.                               Therefore on Wednesday, January 17th 2024, your current subscription for nixos.ems.host homeserver will come to an end. | What happens next? | If you want to keep your service active, you can either: | 1. Upgrade to an Element Business subscription with reduced minimum user commitments (see below) | 2. Or, if you have a custom domain, you can download a copy of your data and migrate to another Matrix hosting provider (or host your own homeserver) | Alternatively, if you are no longer using the service, you can proactively cancel your subscription or do                               nothing and your server will be automatically decommissioned on Wednesday, January 17th 2024. | For existing customers who wish to upgrade to an Element Business subscription, we are exceptionally offering a reduced minimum user requirement of 20 users (usually 50 users) at $5 per user, per month (excluding tax).                               This would have to be paid annually in advance. The plan includes everything covered by your previous                               subscription, plus other features such as single sign-on support. For more information about Element                               Business and our other plans please visit our website. | If you would like to upgrade your hosting plan, please contact our Support team before Wednesday, January 17th 2024                               through our web portal. | To migrate your data to another provider or host your own homeserver, please see our documentation about migrating away from Element. | If we do not hear from you by Wednesday, January 17th 2024, we will cancel your subscription for nixos.ems.host,                               along with any additional subscriptions linked to this host (e.g. bridges), shut down your service and                               delete all associated data. You can also cancel your subscription before the termination date via our web portal. | We are truly sorry for any inconvenience this causes, unfortunately we can’t support these services anymore.                               You can find alternative Matrix hosting providers on the Matrix website. | If you have any questions on the information above please don’t hesitate to contact our Support team, ems-support@element.io. | Thanks,                               The Element Team
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
Dear Graham Christensen,
This is the second reminder regarding the communication we have previously sent.                               We are writing to inform you we are discontinuing our consumer hosting products and legacy plans.                               Therefore on Wednesday, January 17th 2024, your current subscription for nixos.ems.host homeserver will come to an end.
What happens next?
If you want to keep your service active, you can either:
1. Upgrade to an Element Business subscription with reduced minimum user commitments (see below)
2. Or, if you have a custom domain, you can download a copy of your data and migrate to another Matrix hosting provider (or host your own homeserver)
Alternatively, if you are no longer using the service, you can proactively cancel your subscription or do                               nothing and your server will be automatically decommissioned on Wednesday, January 17th 2024.
For existing customers who wish to upgrade to an Element Business subscription, we are exceptionally offering a reduced minimum user requirement of 20 users (usually 50 users) at $5 per user, per month (excluding tax).                               This would have to be paid annually in advance. The plan includes everything covered by your previous                               subscription, plus other features such as single sign-on support. For more information about Element                               Business and our other plans please visit our website.
If you would like to upgrade your hosting plan, please contact our Support team before Wednesday, January 17th 2024                               through our web portal.
To migrate your data to another provider or host your own homeserver, please see our documentation about migrating away from Element.
If we do not hear from you by Wednesday, January 17th 2024, we will cancel your subscription for nixos.ems.host,                               along with any additional subscriptions linked to this host (e.g. bridges), shut down your service and                               delete all associated data. You can also cancel your subscription before the termination date via our web portal.
We are truly sorry for any inconvenience this causes, unfortunately we can’t support these services anymore.                               You can find alternative Matrix hosting providers on the Matrix website.
If you have any questions on the information above please don’t hesitate to contact our Support team, ems-support@element.io.
Thanks,                               The Element Team

Dear Graham Christensen, 
This is the second reminder regarding the communication we have previously sent. We are writing to inform you we are discontinuing our consumer hosting products and legacy plans. Therefore on Wednesday, January 17th 2024, your current subscription for nixos.ems.host homeserver will come to an end. 
What happens next? 
If you want to keep your service active, you can either: 
1. Upgrade to an Element Business subscription with reduced minimum user commitments (see below)
2. Or, if you have a custom domain, you can download a copy of your data and migrate to [another Matrix hosting provider](https://www.matrix.org/ecosystem/hosting/) (or host your own [homeserver](https://www.matrix.org/ecosystem/servers/))
Alternatively, if you are no longer using the service, you can proactively cancel your subscription or do nothing and your server will be automatically decommissioned on Wednesday, January 17th 2024. 
For existing customers who wish to upgrade to an Element Business subscription, we are exceptionally offering a reduced minimum user requirement of 20 users (usually 50 users) at $5 per user, per month (excluding tax). This would have to be paid annually in advance. The plan includes everything covered by your previous subscription, plus other features such as single sign-on support. For more information about Element Business and our other plans please visit our [website](https://element.io/pricing). 
If you would like to upgrade your hosting plan, please contact our Support team before Wednesday, January 17th 2024 through our [web portal](https://ems.element.io/support). 
To migrate your data to another provider or host your own homeserver, please see our [documentation](https://ems-docs.element.io/books/element-cloud-documentation/page/migrate-from-ems-to-self-hosted) about migrating away from Element. 
If we do not hear from you by Wednesday, January 17th 2024, we will cancel your subscription for nixos.ems.host, along with any additional subscriptions linked to this host (e.g. bridges), shut down your service and delete all associated data. You can also cancel your subscription before the termination date via our [web portal](https://ems.element.io/user/hosting). 
We are truly sorry for any inconvenience this causes, unfortunately we can’t support these services anymore. You can find alternative Matrix hosting providers on the [Matrix website](https://www.matrix.org/ecosystem/hosting/). 
If you have any questions on the information above please don’t hesitate to contact our Support team, [ems-support@element.io](mailto:ems-support@element.io). 
Thanks, 
The Element Team

delroth commented 8 months ago

Thank you! Can you please follow up here with the response as soon as you get it? This is even shorter notice than I expected, so we need to make a plan quickly.

In the meantime, when you have a chance, could you clarify exactly who currently has access to the EMS account? If it's only you and/or Eelco, would you mind granting @zimbatm access as well so we don't have to bother you in the future?

delroth commented 8 months ago

@grahamc Friendly ping as we are less than a week away from irreversible data loss.

delroth commented 8 months ago

We have decided in the infra meeting today that if we have to change to the new $100/month (paid annually plan), we'd rather migrate to self-hosting. @mweinelt will take the lead on setting up the configs for this on one of the non-critical-infra machines.

grahamc commented 8 months ago

Hey folks,

Just an update that the deadline has been extended to February 17 by EMS support. Note that I have a recent backup from the server available, so irreversible data loss is already off the table.

I'm looking into options to produce a backup that isn't loaded with unencrypted private messages.

grahamc commented 8 months ago

Another thing is that since the server doesn't host any active users (...anymore... sigh, the convenience of using the account personally and for configuration was a mistake ...) almost no unique data is stored by the server itself. The server primarily acts as a map of room hashes to the room's name. This data can be easily backed up by ~anyone.

delroth commented 8 months ago

In theory I agree with you, in practice I'm not looking forward to the amount of surgery that would be needed to transplant that data into a fresh server - and it's not immediately obvious to me whether there would be anything bound to e.g. crypto keys we'd have no way to have continuity on without a "real" backup.

Since your old :nixos.org account isn't used anymore doesn't EMS support some kind of full Matrix account deletion/purge? I don't see how they could get away without it in the EU due to GDPR, so maybe the easiest way would be to purge all your personal data and then export a new backup.

grahamc commented 8 months ago

In theory I agree with you, in practice [...]

Yeah, absolutely. Like I said, I do have an actual database dump :).

Since your old :nixos.org account isn't used anymore doesn't EMS support some kind of full Matrix account deletion/purge?

I've already exercised that option, and what remains is still ~12k uploads and many gigabytes of messages and other events, lots with raw text.

zimbatm commented 8 months ago

@grahamc is it possible to share the EMS password so we can handle the failover ourselves? Given how busy you are, it would make things easier.

grahamc commented 8 months ago

I don't want to do that as-is because of the significant amount of private information of mine and others (people I talked to) available unencrypted in the database. I'm working to remove more of my information before sending an export.

zimbatm commented 8 months ago

Ok. I assumed that DMs would be end-to-end encrypted by default in Matrix? The EMS and Matrix user account password are two different entities AFAIK.

grahamc commented 8 months ago

I assumed that too. Most of them are, some are not. What remains are thousands of messages and uploads -- available unencrypted in the database.

zimbatm commented 8 months ago

@grahamc we need a bit of buffer so the infra team can migrate and test things on a reasonable timeline. Can you commit to releasing the data at least two weeks prior deadline? That means next weekend.

Alternatively, DetSys could sponsor the EMS hosting.

Janik-Haag commented 7 months ago

IIRC @RaitoBezarius talked with people working at element about this at fosdem?

zimbatm commented 7 months ago

Ron was going to talk to them to see if we could extend the deadline.

But we also need to get a sense from Graham to know how much extra time is needed. Or maybe there is something we can do to help the process.

In theory EMS should respond to GDPR requests and have tooling available to delete user data.

grahamc commented 7 months ago

I've been in contact with EMS, and have offered to pay for the necessary tier to keep it running beyond the deadline. However, they've brought the CEO and VP of Eng into the conversation to extract the necessary information. At this point I would not expect the service to go away on the 17th, since I have offered to pay for it. And, obviously I'm looking to get and transfer the data from the EMS team as soon as they can provide it.

RaitoBezarius commented 7 months ago

@grahamc Thank you for the update, we entered into contact with the executives during FOSDEM as we wanted to see if they could apply the user removal process and make your work load lower on this problem, I would imagine their offer can still stand if you would prefer not to pay.

grahamc commented 7 months ago

Right, we've been discussing that in my email thread over the past weeks. I of course don't relish that this situation exists in the first place, and paying for it is just delaying a real solution. So of course it is my preference to not need to pay (though I don't mind, and have been glad to since the migration to Matrix), and for the data transfer to be clean and straightforward. And if it can't be done in the timeline EMS would need to do their work, I will pay for it.

refroni commented 7 months ago

Quick note here as additional context: I asked Amandine (COO & Co-Founder at Matrix) to help us out and she met with us at Fosdem with their VP of Eng Neil on Sunday. @grahamc is there anything else we can do to help?

grahamc commented 7 months ago

Just got this mail from EMS:

As a result of this additional option, your host nixos.ems.host has not automatically deprovisioned, we have extended the deadline to Thursday, February 22nd 2024 and no further payments have or will be taken.

grahamc commented 7 months ago

I heard yesterday that Neil at EMS has asked the team to pause decommissioning of the instance while we work to solve the data issue.

NixOS / infra

Migrate off EMS to self-hosted Synapse #325