Process for security patches

[robbkidd]

Dance Madly on the Lip of a Volcano [I] - Jess Frazelle & Brandon Philips

[nellshamrell]

Damn, that talk is pure gold!

Here are my notes on it:

• “Dancing on the Edge of a Volcano" by Jess Frazelle and Brandon Philips
• https://www.youtube.com/watch?v=sNjylW8FV9A&feature=youtu.be

Context Setting

• Teetering on edge of a volcano

• one side - might fall in b/c of security vulnerability

• other side - might fall b/c don't have processes for dealing with security vulnerability

• tons of users on the internet

• everyone's data is moving onto servers - documents, commerce, communications

Kubernetes

• designed to maintain LOTS of servers

Open Source and Security Disclosure

• want an open process
• need a way that people won't find out about the bugs and use them in a bad way
• need way to let vendors know and not break embargo
• can use GPG
• can you make everyone happy?

Users Expectations

• want software without bugs
• want to be able to upgrade without breaking changes
• want to be let know when security fixes are made so they can act on them accordingly

Original Reporter Expectations

• Want "a human to say acknowledged"
• Want update on status transparency
• Want bug to be fixed in a certain timeframe

Project Expectations

• want bugs to be fixed quickly and embargo to be kept secret until time of disclosure
• want upgrades to remain seamless
• want to continue to instill trust with users

How to get poeple to upgrade?

• challenges with making sure people know
• use mailing lists, establish a pattern for the project that allows people to follow along
• how to handle if very severe...pray you don't have someone make up a fun name

How bugs get reported

• sometimes found by a researcher - looks up process for reporting
• sometimes don't realize severity, file issue on main open source repo

Mistakes happen

• not everyone can recognize the severity of a bug at first glance
• allow for mistakes and have a process for handling them in a way that it will not confuse the original reporter or cause unnecessary attention

Learning from other communities

Common Stuff

• Make documentation Googleable (X security disclosure)
• Have a dedicated response team
• Coordinate via a mailing list - don't force use of GPG
• Have internal documentation on the process
• Use a severity leveling system
• Request CVEs when appropriate

Docker

• Document Everything - doing a security release should be EASY
• Tooling for Everything - all scripts to build and release Docker can be run by anyone at any time

Linux

• Don't negotiate
• Set timeline expectations
• don't negotiate on embargos

Node JS

• Help the ecosystem
• has a "core" security team AND a community security team

Go

• Early warning system
• give people a heads up that a vulnerability is coming
• small team, explicit contacts
• mailing lists fail, people don't read email, list everyone (list specific people)

Kubernetes Security Process

Normal Release Process

• at any given point, have three minor releases for Kubnernetes being prepared at any given time
• every minor release scheduled
• have a release team for each release

Security Release Process

1. First, bug is reported through disclosure process
2. Within 24 hours fix team is summoned/formed
3. Fix Development (on private repo that is set up)
4. Disclosure of "Fix forthcoming" to users
5. Patch Disclosure to distributions (i.e. vendors)
6. Release Day (within 21 days) - code mervged from private repo back into public repo

Scoring Issues with CVSS

• Common Vulnerability Scoring System Calculator

Areas for Improvement

Rotating the Responsibilities

• Challenges
• Responsible security process requires non-public work
• Running the process is not glamorous
• Incentives
• Many organizations rely on Kubernetes, the work is important
• Critical security announcements will be highly referened
• Next Steps
• We need a way of making decisions! Kubernetes Governance Discussion concludes

Communication to Users

• Send an email to kubernetes-announce and call it a day?!
• How do you make announcements immediately actionable?
• Should vendors be informed ahead of time with a fix?

[robbkidd]

Hey, look! We've already got a security policy and submission process!

It shows up as the first result when I googled "chef security disclosure". 👍

Reviewing it, I think it has most of what we want to communicate publicly about the process. Things we should consider in addition to what is already there:

• update the public page to state that we do not have a vulnerability bounty program
• let submitters rub some crypto on their submissions?
• establish a severity level assessment/scoring model
• to respond in a timely but humane way, do more humans need to be empowered to make the initial response?
• some canned responses
• document the internal security fix process so teams don't have to make it up when it happens to them for the first time, e.g.:
  • define communication paths to downstream projects (tentative)
  • track the work privately (Jira? issue in a private GitHub fork?)
  • work in a private fork
  • test in a private environment (no public test results until after fix is available)
  • decide on a timeline for public notice that a fix will be released
  • make the announcement that a fix will be released on a specific date
  • release the ~kraken~ fix and make release announcement

[benr]

Some thoughts in response to the discussion already:

• Indeed, https://www.chef.io/security/ is already in place
• More clarity about a lack of Bug Bounties is a good idea
• Vulnerability scoring should be done according to the CVSS standard (EDIT by Robb to add a link to the standard)
• Process for validating, assessing, remediation, and resolving vulnerabilities should be enhanced and standardized against all teams; Ideally this would correspond with enhancements in our automated scanning and testing. Refer to my recommendations here: https://docs.google.com/document/d/1O9MFOz6hSqV_-YfggLcsQtqGDD9Y-2vGoBVU5sV214g/edit?usp=sharing
• The stock answer to "how to communicate" has been "The Blog"; beyond this it is at the discretion of the responding team based on the scope and impact of the issue.
• Most of all; we must have SLO's around the process and ensure our Engineering process accepts these objectives

[nellshamrell]

Done with #101