Expired @openobservatory.org

[darkk]

Impact:

• mailboxes were unaccessible for 4...5 hours
• some incoming mail was rejected (bounced)

Detection: alert from user

Timeline UTC, 04 Oct: 09:28 @darkk fetchmail OK 10:38 @darkk fetchmail failed 11:53 @agrabeli: For some reason I can't login to my email 12:27 @darkk: Account expired 12:39 twitter ranting, @darkk assumed that expiration notifications were broken 13:14 @darkk mails to support@runbox.com 13:27 ticked created, ticket ID: JVG-692-95923 14:00 @joelanders I can still get my Runbox inbox 14:17 Dave from Runbox: What I have done is extend the expiry period until today. What that does is give you the 2 weeks grace period again until payment is due. So what should happen is the account close on 18-Oct-2017. It means the Main account will probably still get expiry notifications. The accounts are open again and will start to receive mail in the next 15 minutes or so. I will check they remain open tomorrow so that I know that my plan is working.

What went well:

• Runbox good will

What went wrong:

• @hellais (main account holder) was flying and usual procedure was impossible.
• As Catalonia incident is ongoing, @darkk assumes that some important messages could be lost. Messages sent to closed accounts are bounced back to sender.
• Expiration notification were lost several times a row:

It is the responsibility of the Main account holder to renew sub-accounts but I will see if I can come up with a workaround. Notifications of expiry would have been sent multiple times starting around 6 weeks ago. The account would have technically expired 2 weeks ago and then we have a 2 week grace period so that any payment issues can be sorted out. So we are quite generous with the time periods. Let me see what I can do in this exceptional case. — Dave from Runbox

What could be done to prevent relapse and decrease impact:

• [x] make Main account a shared account to decrease bus-factor, e.g. share password among two persons, or set recovery procedure with all team@ or admin@ having password shared generated with ssss.
• [x] move to another email service provider or hosted solution (IMHO that's overreacting)
• [x] what other "shared" accounts do we need? AWS?... namecheap?...

[darkk]

what other "shared" accounts do we need? AWS?... namecheap?...

We brainstormed Inventory of Single Points of Failure and found more than dozen. That's good to have them explicitly listed :)

[darkk]

@openobservatory.org is moved to G Suite. Also, now we maintain list of SPoFs and review it periodically. That's a good exercise to have a failover plan written down.