This is the repo for swyx's blog - Blog content is created in github issues, then posted on swyx.io as blog pages! Comment/watch to follow along my blog within GitHub
Today's fun emergency at work was a first - writing a security postmortem for a breach of an open source user (aka not a breach of us, which seems the norm).
You can see the final result here. The total turnaround time for this post, from deciding to write something, to having it live, was about 4 hours, including review from about 6 team members.
It's not the most gripping piece of literature I've ever penned, but it was a muscle I exercise rarely so I figured I would share some thoughts if you should ever need to do the same.
The opening
I prioritized putting the bottom line up front. In the first sentence, og:description, and on social media, you learn the basic facts of the matter: "an Airbyte user self-hosting an unsecured instance had their connector credentials stolen". This is enough for most people to get the gist. I hate when companies bury the lede - especially in our case when people were already talking about the incident in our Slack and screenshots were being shared on social media. Getting the story out in the first sentence also lets people move on with their day if that's all the time they had to spend with us.
This leads me to the next point: there are multiple audiences for any security blogpost.
Unsophisticated people whose first animal instinct at the news of every hack is "uh oh, these guys aren't secure",
to more familiar users who would never be caught dead making the same mistakes,
all the way through the other side to superexpert users who have seen 100 similar mistakes, and, while individually excusable, as a whole bemoan the sorry state of security in our industry.
I ordered my message for the outermost rings first, who have the least attention span, and then focused on reassuring existing users, who care about the details.
It was important to thread the needle of:
not placing the entire blame on the user
reassuring how seriously we take it, with proof
answering the jabs of our critics
not hard-selling our managed/hosted Cloud solution to open source users
This was sometimes a tricky thing - I hesitated a lot before landing on the AWS-like messaging of Shared Responsibility (hey that media training was worth something after all!). In the end, I had one reviewer each that felt we were taking too much blame and pushing too much blame in different parts of the post, and that felt about as good as I could get.
A last minute review by a new hire reminded us that our transparency was one of the reasons he joined us, and I thought that sounded like a pretty good description of what we were doing, so that went up as the second sentence of the first para: "Transparency is a core value at Airbyte, so we are choosing to highlight this to our community and discuss the steps we will take to improve Airbyte's security defaults." - in other words, we could've just not said anything about this issue, but chose to bring attention to it anyway since
it was already being discussed by the extremely online types and
it might have benefit for other users who were out of the loop.
Presenting facts
A chronology with referencable links and version numbers adds credibility - nothing to hide. Giving people the facts as we would if we were talking in person helps our readers explain it to their colleagues as well.
Followups
After presenting the "what happened", we then laid out:
"Action Items": what you should do and what we will do
"FAQs": our direct responses to criticisms we saw elsewhere, that readers might be thinking, that we answered directly
Part of the tricky part of Action Items is that Product/Engineering have to agree with you on what we remedial actions we are willing to commit to. This is tricky particularly if you are not really admitting fault or taking full responsibility. Security isn't binary - particularly when discussing security of defaults - there is an unending menu of increasingly secure things we could opt to do, all with tradeoffs of eng time and open source convenience/infrastructure requirements. Fortunately our engineering team either had already planned some of these actions or were more than willing to add them onto the roadmap.
The final part of handling FAQs was broadening the thinking to proactively answer questions that hadn't yet been asked but that you might reasonably expect to see in a post like this. Hence putting up a responsible disclosure policy, and phrasing the Airbyte Cloud question more like a cynic might.
Today's fun emergency at work was a first - writing a security postmortem for a breach of an open source user (aka not a breach of us, which seems the norm).
You can see the final result here. The total turnaround time for this post, from deciding to write something, to having it live, was about 4 hours, including review from about 6 team members.
It's not the most gripping piece of literature I've ever penned, but it was a muscle I exercise rarely so I figured I would share some thoughts if you should ever need to do the same.
The opening
Presenting facts
A chronology with referencable links and version numbers adds credibility - nothing to hide. Giving people the facts as we would if we were talking in person helps our readers explain it to their colleagues as well.
Followups
After presenting the "what happened", we then laid out:
Part of the tricky part of Action Items is that Product/Engineering have to agree with you on what we remedial actions we are willing to commit to. This is tricky particularly if you are not really admitting fault or taking full responsibility. Security isn't binary - particularly when discussing security of defaults - there is an unending menu of increasingly secure things we could opt to do, all with tradeoffs of eng time and open source convenience/infrastructure requirements. Fortunately our engineering team either had already planned some of these actions or were more than willing to add them onto the roadmap.
The final part of handling FAQs was broadening the thinking to proactively answer questions that hadn't yet been asked but that you might reasonably expect to see in a post like this. Hence putting up a responsible disclosure policy, and phrasing the Airbyte Cloud question more like a cynic might.