Request for security announce list

[mattfarina]

We have received a request to create a security announce mailing list, similar to the Kubernetes one. That list is for service providers and they are warned 24 hours before the public that there is a security vulnerability.

I'm filing that request here for further discussion.

[DahuK]

@mattfarina Thanks for this proposal! we definitely need it, as service provider we could prepare the customer announcement and fix patch before the public disclosure.

[SteveLasker]

Add an ack for Azure here as well. The biggest question I'd raise is the timing. 24 hours notice, as noted above.

It takes time to understand the impact, design a solution, code it and test it. As cloud providers have multiple regions, it takes time to safely deploy a fix, once it's designed, written and tested. This also assumes the impacted companies drop everything else to work on the issue. Which, we always prioritize production support and security fixes. However, safely deploying a fix is required or "the cure becomes worse than the cold".

The Kubernetes Security and Disclosure Information outlines the timelines as:

Security Vulnerability Response

Each report is acknowledged and analyzed by Product Security Committee members within 3 working days. This will set off the

Security Release Process.

Any vulnerability information shared with Product Security Committee stays within Kubernetes project and will not be disseminated to other projects unless it is necessary to get the issue fixed.

As the security issue moves from triage, to identified fix, to release planning we will keep the reporter updated.

Public Disclosure Timing

A public disclosure date is negotiated by the Kubernetes Product Security Committee and the bug submitter. We prefer to fully disclose the bug as soon as possible once a user mitigation is available. It is reasonable to delay disclosure when the bug or the fix is not yet fully understood, the solution is not well-tested, or for vendor coordination. The timeframe for disclosure is from immediate (especially if it's already publicly known) to a few weeks. For a vulnerability with a straightforward mitigation, we expect report date to disclosure date to be on the order of 7 days. The Kubernetes Product Security Committee holds the final say when setting a disclosure date.

I'm not a security expert, and I try not to play one on TV, (the internet). I'd highly recommend engaging security experts from CNCF, aws, azure and google, amongst others that can provide experience on how to best handle a vulnerability notification process.

[mattfarina]

For what its worth, Helm has a security team and process. What is missing, that Kubernetes has, is the vendor embargo private distributors list. Most CNCF projects do not have this.

Now that we have had more than 3 requests for the embargo list we a looking to set that up. It will likely follow what k8s and Harbor have done.

@SteveLasker In the k8s private distributors list member criteria there is an item of "Not be a downstream". This is one item we'd need to work out for Helm as something like ACR is downstream. ACR was not only a downstream project but had also changed the output of the index.yaml file to add custom fields outside the built-in customizable location. In that capacity, it's a bit of a fork.

I expect the Helm list to operate a little different from k8s when it comes to downstream projects from what k8s does. But, I also expect that a forked experience from Helm is something that will be taken into account.

[SteveLasker]

Hi Matt, it's great to hear that there is going to be something done for security related notifications so that large-scale users of Helm can validate the impact prior to release of well-intended changes. Helm is a huge production deployment effort for Kubernetes -- the other huge project. Having a proactive security solution will give confidence to customers wanting to take a dependency on Helm for their critical production workloads and we’re happy to participate in helping that effort.

I don't, however, understand the references to ACR being a "fork". ACR doesn’t implement any Helm code, rather we accept and serve helm charts, providing a Helm index.yaml for local chart evaluation. As mentioned previously, this was implemented in collaboration with yourself and the Chart Museum maintainers.

Can you point to where the helm index.yaml is defined as the Helm spec is empty? We implemented what we found in other examples and what worked for the last 2 years. As a reminder, it wasn’t just ACR that had issues, but several other charts and implementations of Chart Museum as noted here:

This is unfortunately one of the side effects of one of the security fixes. In order to address one of the security issues, Helm now has to validate that the index.yaml file does not contain any extra fields, which chartmuseum does appear to add.

Can you clarify the comment about not willing to notify Azure of security?

This is one item we'd need to work out for Helm as something like ACR is downstream. ACR was not only a downstream project but had also changed the output of the index.yaml file to add custom fields outside the built-in customizable location. I expect the Helm list to operate a little different from k8s when it comes to downstream projects from what k8s does. But, I also expect that a forked experience from Helm is something that will be taken into account.

It would be helpful for the Helm maintainers to outline its relationship with cloud providers that want to support the use of Helm for production critical workloads. This is the model k8s is engaging and I'd suggest Helm as a production deployment technology would want to support a similar process.

Helm, through OCI Artifacts, is now supported in all major cloud providers, including projects like Harbor.. GitHub and others are adding support as well. This all started with the desire to lift Helm into the production secured workflows enabling a generalized RegOps workflow.

We recognize well-intended issues will surface. We're just asking what is being done to assure we're all learning and putting in place new safeguards to help our shared users be secure and productive.