Security Audit for Kubernetes on behalf of CNCF

caniszczyk commented 6 years ago

The CNCF has been piloting a security audit program for CNCF projects:

https://coredns.io/2018/03/15/cure53-security-assessment/ https://github.com/envoyproxy/envoy#security-audit https://cure53.de/pentest-report_prometheus.pdf

The pilot has been successful and we are happy to start offering this to other CNCF projects that are interested. We would give preference to graduated projects first. After speaking with @bgrant0607 a bit he suggested making this a KSC topic, so here we are.

cblecker commented 6 years ago

cc PST: @philips @jessfraz @cjcullen @tallclair @liggitt

jessfraz commented 6 years ago

can we get a company that has a proven track record like NCC group or something... ping @dyn- @killahertz who did the docker one, but no longer work there and may know of good people...

EDIT: I don't think that's jesse's handle but I can't find his...

jessfraz commented 6 years ago

ah no it's @jhertz , sorry for the noise

jessfraz commented 6 years ago

also @tqbf and @ddz who might know good people

(I would like to find a legit company otherwise it's not worth it, not that that other company is not legit just I have not heard of it... someone correct me if I'm wrong... :)

jessfraz commented 6 years ago

I would like to just put for comparison sake the quality of the work from NCC group, versus the links Chris gave above... I looked them over...

https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2016/june/container_whitepaper.pdf https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2016/april/ncc_group_understanding_hardening_linux_containers-1-1.pdf

caniszczyk commented 6 years ago

happy to use whatever vendor the KSC prefers, CNCF is happy to make it happen

On Aug 27, 2018, at 8:34 PM, Jess Frazelle notifications@github.com wrote:

I would like to just put for comparison sake the quality of the work from NCC group, versus the links Chris gave above... I looked them over...

https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2016/june/container_whitepaper.pdf https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2016/april/ncc_group_understanding_hardening_linux_containers-1-1.pdf

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

ddz commented 6 years ago

Cure53 is actually very good. They have been very good about publishing their audit reports of open source projects (not many security companies do this) and their findings in the linked reports look very solid to me.

jessfraz commented 6 years ago

Thanks @ddz that's super helpful and gives me a lot more confidence :)

timothysc commented 6 years ago

/cc @mattmoyer

philips commented 6 years ago

I have only helped lead two third party security audits in my career but I think the process for selection should be something like this:

Customer Audit Team

Form a team of 2-3 Kubernetes people to lead the process (audit leads)
Audit leads create a list of 5-10 focus areas (likely based on the bug bounty scope cc @tallclair)
Identify 5-10 kube contributors on a subject matter experts (SMEs) on the focus areas

Evaluate Vendor Teams

Identify 2-3 vendors to evaluate
Get resumes/CVs/reports of the lead and supporting audit team (I think this is important because with all due respect to the vendors the makeup of the team is more important than the brand of the vendor)
Let the audit leads choose the vendor based on their past experience and the desired focus areas.

Work with Vendor

Establish a kickoff meeting where the audit leads train the vendor team on the product architecture, build process, and getting things running
Establish regular weekly or bi-weekly sync during engagement between audit leads and vendor
Engage SMEs as necessary to provide context and help guide the vendor
Disclose urgent issues to the existing security@kubernetes.io processes

Report Out

The audit leads should do review rounds on drafts of the report along with the SMEs, and Kubernetes Product Security Team providing feedback to the vendor
Audit leads work with PST to decide on responsible day to open the final draft up publicly

If people think this is roughly right I can put together a Google Doc to work on.

philips commented 6 years ago

FWIW, I have worked with Matasano, NCC, and received a recommendation for Cure53 from someone I trust.

That said I deeply believe it is the individuals and not the vendor brand that matters and their fit to the engagement.

tqbf commented 6 years ago

I've run assessments for more than a decade and the process @philips outlines is one of the smarter client processes I've seen, and I strongly recommend you all consider it. In particular: having clear focus areas you want evaluated, and a team ready to walk the testers through the steps of using k8s in those focus areas.

What I don't think you'll get much value from is hiring a team of assessors to just sort of read through your code looking for scary things that stick out. I also think the Venn diagram of "good at finding software vulns" and "understands how k8s is commonly used" is not that great, so again: really like the idea of being ready to handhold assessment team through actual usage.

jessfraz commented 6 years ago

This all sounds great to me and I agree it's the individual researchers who matter not vendor :)

philips commented 6 years ago

Steering committee: would you mind delegating vendor selection responsibility to a 2-3 person audit team selected by the PST? It might not be people from the PST but we could help find the right folks.

On Tue, Aug 28, 2018 at 3:52 PM Jess Frazelle notifications@github.com wrote:

This all sounds great to me and I agree it's the individual researchers who matter not vendor :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/steering/issues/71#issuecomment-416766254, or mute the thread https://github.com/notifications/unsubscribe-auth/AACDCHOvV3dJKa509fb047mjajL17v0Nks5uVcmsgaJpZM4WOuYf .

joelsmith commented 6 years ago

We used Insomnia Security for an engagement specific to OpenShift and they did a fantastic job. They uncovered privilege escalation vector in our build system and made lots of other recommendations. They have a working knowledge of k8s owing to the work they did for Red Hat. I can provide the name of the lead researcher upon request.

philips commented 6 years ago

Through lazy consensus the steering committee has delegated forming a security audit team to the PST.

I think the next step is to solicit volunteers to run the audit from folks on SIG Auth and the PST. Relevant questions:

What experience do you have with a third party security audit?
What experience do you have with the subsystems of Kubernetes that will likely be in scope based on the bug bounty definition draft?
Will you have support of your employer to spend about 8-12 hours in total over 4-6 weeks in support of this audit?

philips commented 6 years ago

Moving discussion to SIG Release Issue https://github.com/kubernetes/sig-release/issues/284

philips commented 5 years ago

Thank you to everyone who volunteered to help with this effort! We really appreciate it.

The PST has selected Jess Frazelle @jessfraz, Joel Smith @joelsmith, and Aaron Small @aasmall to lead this effort. These leaders will have final call on vendor interview/coaching schedules, vendor selection, and the community report.

That said everyone is encouraged to participate. We expect the folks above to be transparent about meeting times, and process. I expect that they will form a working group to coordinate and announce that working group to SIG Auth, the PST, and the Steering committee.

philips commented 5 years ago

The team had their first meeting: https://groups.google.com/forum/#!topic/kubernetes-sig-auth/DOHShVCW6RM

joelsmith commented 5 years ago

FYI, @jessfraz has stepped down as one of the wg leaders since she's too busy now, and the PST has selected Craig Ingram @cji to replace her.

miskun commented 5 years ago

So, is the plan to limit the scope on K8S "kernel" only or is there plan to provide security audit also for more complete solutions? I'm talking about K8S distros. If so, we'd like to be involved (Kontena Pharos distro).

aasmall commented 5 years ago

It's been a while since we've updated this larger audience.

We published our RFP and invited several vendors to participate on 2018/10/29.
Between 2018/10/29 and 2018/11/12 we worked with the vendors to answer any questions they had. (They had lots of good ones)
The RFP closed on 2018/11/3 and since then we've been reviewing their proposals.
We hope to have a vendor selected by next week.

It's worth noting that we probably won't be able to actually begin the audit until early next year as December is both a busy month for the industry and a lot of people will be on leave.

We want these vendors to bring their best researchers, so we're willing to wait.

aasmall commented 5 years ago

So, is the plan to limit the scope on K8S "kernel" only or is there plan to provide security audit also for more complete solutions? I'm talking about K8S distros. If so, we'd like to be involved (Kontena Pharos distro).

@miskun Right now, we are scoping the audit to core kubernetes, as defined in the bug bounty program. Even scoping it that tightly is a huge undertaking, and I think diluting it would really hurt the final product.

I'm definitly open to discussion if that was not the original intent of the committee.

philips commented 5 years ago

I am going to close this out since it has been delegated to https://github.com/kubernetes/community/tree/master/wg-security-audit and the work is underway.

/close

k8s-ci-robot commented 5 years ago

@philips: Closing this issue.

In response to [this](https://github.com/kubernetes/steering/issues/71#issuecomment-454885144): >I am going to close this out since it has been delegated to https://github.com/kubernetes/community/tree/master/wg-security-audit and the work is underway. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

kubernetes / steering