Open jesseallhands opened 2 years ago
@lucaslcode @aapeliv @ShmuelTreiger @darrenvong This is a critical issue affecting nearly all areas of the website and affecting nearly all functionality of the website!
Issue was that we ran out of disk space. Lucas expanded space as a temporary solution. Thus the issue seems to be resolved as of 01:54 UTC. A postmortem is still needed.
Can we close this? Or are we waiting until the post-mortem has been done first?
I was waiting for the postmortem, but maybe that should be it's own ticket since we don't really have a postmortem protocol yet?
The reason a postmortem is so important in this case is that this outage seems to have been fully preventable. If the issue was exceeding a storage limit, we should have had an automated alarm that notified us well in advance of us hitting the limit. If there was an alarm that just didn't work, we should find out why.
The reason a postmortem is so important in this case is that this outage seems to have been fully preventable. If the issue was exceeding a storage limit, we should have had an automated alarm that notified us well in advance of us hitting the limit. If there was an alarm that just didn't work, we should find out why.
I agree with everything that you said there - it's all about learning from this and see if we can do better next time to refine our processes!
And I feel like maybe it should be a separate issue, as post-mortems tend to be documented as something we can refer back to in the future and this repo tends to only have docs related to how the code works. But we also have meeting notes in a docs
folder so could be argued that postmortems could be here too 🤷
Although things might be better structured/easier to find if it's saved on our wiki instead? Devs (certainly myself) tend to use GitHub a lot more though, so I don't have any objections of them being in docs
either. Would be interested to hear what others think too @aapeliv @lucaslcode ?
Yes, plan has been to write postmortems; but we had no meaningful outages so we forgot about it!
This seems to be a big issue. You cannot search nearly anything or bring up profiles at all. Seems that most APIs are broken: