[Proposal] Apply Supply Chain Security WG Best Practices to new or existing CNCF Project

[mlieberman85]

Description: Apply best practices as defined by the Supply Chain Security WG's Best Practices guide as well as any additional practices as defined in the Secure Software Factory ref arch.

Impact: This will help both end users as well as the CNCF itself. One goal is to help end users better understand how to actually apply the best practices. Another goal is to help solve some of the supply chain challenges we have in general by dogfooding our own work and applying the best practices to one or more CNCF projects.

Scope: The primary scope of this proposal is to both apply best practices to a project as well as highlight any challenges or gaps in tooling that would prevent a project from adopting all of the best practices. The amount of time this could take is highly dependent on the size and scope of the project we would be applying the practices to. If this is a new small project this probably wouldn't take very much time. If we were to apply this to a larger already established project this could take a lot of effort due to the inertia in changing established processes.

Note: discussing possible collaboration with Kubernetes sig-release on May 31 11:30 AM eastern

TO DO

• [ ] Security TAG Leadership Representative:
• [ ] Project leader(s):
• [ ] Project Members:
• [ ] Fill in addition TODO items here so the project team and community can see progress!
• [ ] Scope
• [ ] Deliverable(s)
• [ ] Project Schedule
• [ ] Slack Channel (as needed)
• [ ] Meeting Time & Day:
• [ ] Meeting Notes (link)
• [ ] Meeting Details (zoom or hangouts link)
• [ ] Retrospective

[pxp928]

There has been discussion with that cert-manager might be interested in going down this route. They were looking for help with a security audit and how they can meet the SLSA levels. Need to have more discussions with the cert-manager team. Also might require some prep work as they are currently using bazel to build.

[lumjjb]

+1 think this could be an interesting exercise and could scale this through requesting some funding!

[JustinCappos]

We'd be happy to help in a few different ways.

1. Support efforts to adopt tools from communities we're engaged with like in-toto, TUF, reproducible builds, etc.
2. Pilot this ourselves in some of our projects that aren't going through all the steps.
3. Gather data on experiences from other groups and write up some community guidelines, etc. so that people understand the likely costs and stumbling blocks.

[TheFoxAtWork]

I just had a great conversation with @puerco about the Kubernetes SIG-Release process. An excellent way we can get started in this is by applying the supply chain best practices to the Kubernetes Build Systems. They’ve done a lot of work applying best practices to K8s itself, but the build processes don’t have the same love (this is a lot of work). I’ve talked about this as well with a few folks in LF/CNCF staff and I’d like us to coordinate with the CNCF Staff for the compute resources and new staff (such as a technical program manager) to help on this. It will require us to collaborate closely with Kubernetes SIG-Security and SIG-Release.

As a first effort to roll this out on, the build system improvements can be done in parallel to the existing work SIG-Release has done. Without impacting Kubernetes until we’ve tested the build process for the build images, libraries, and packages meet the needs of Kubernetes. From there we can partner again to adjust the Kubernetes release process to take in those changes into verification.

[PushkarJ]

With my both hats on from TAG Security and SIG Security -- let's do this and let me know where I can help !! 🎉

[lumjjb]

+10000! Exciting!

[mlieberman85]

@TheFoxAtWork I think it would be worthwhile to chat this over with everyone once folks are back from Kubecon EU. We initially avoided Kubernetes specific work because of how large Kubernetes' project is compared to let's say cert-manager.

My worry is we take on a lot of heavy work while still sorting out some of the solutions compared with starting small and moving up. With that said if there's already a bunch of work moving in that area already it could be fine.

[TheFoxAtWork]

+1 to the discussion. Based on what was shared today, we may have a good starting point.

[ragashreeshekar]

I'd be interested to contribute here.

[mlieberman85]

@TheFoxAtWork Now with folks back from Valencia, when might people want to have a chat about some details. We still have the meetings weekly at 11AM Eastern time for the supply chain working group. I am also down to talk through some of the work that is being done and logistic and whatnot outside of that as well.

[hectorj2f]

+1 I'd be interested on participating too.

[TheFoxAtWork]

@mlieberman85 I'm not free for the next 3 meetings :(. I don't want to be the hold up for this happening:

• A more in-depth presentation/discussion with SIG Release to learn more about their overall build process for Kubernetes and a deeper focus on how they build their build environments and images for their release process (this is the proposed focus area)
• Evaluation of what is current already in place for their build image creation compared to best practices.
• Roughly scope the level of effort to determine: what minimally could be done, what could ideally be done, what is the extent of changes for the pre-existing build process, what needs modified versus newly integrated?
• Decide on a go/no-go for the project (dependent on scope, reproducibility to other projects leveraging similar build pipelines, resource and technical talent availability, interest from SIG-Release to pursue)

[mlieberman85]

Cool. As part of the analysis around this, I created a few graph visualizations (difficult to read I know, took overnight to render) to just show how complicated the supply chain graph is when taking into account all the things involved in both the runtime and build process. There are approximately 650 packages involved

In fact the images are so large they won't fit in this issue (the pngs are 50+ megs) .. I'll try and upload it somewhere and link here.

[mlieberman85]

Here is a 7-ish mb SVG of the graph. It struggles to render.... It also is a graph PURELY of the non-go dependencies. The idea here is not to be actually a good representation of the supply chain problem outside of just being a shock. With some of the tooling in OpenSSF (Sigstore, Frsca) and some other stuff I've built myself we can do some interesting analysis on the supply chain graph to maybe find where there is some easy ROI in addition to the internal to k8s supply chain.

I can build up a bigger proposal here but the main things I was looking at was:

• Securing internal supply chain of k8s. These are the projects and source code underneath the ownership of the k8s maintainers/CNCF. This would be helping secure the build, apply best practices, help them use a "secure software factory" that is aligned with the ref arch.
• Securing external supply chain of k8s. Find where there could be some blind spots around what k8s itself relies on. Is there some package in a critical path that is not well maintained? Are there any suspicious packages in the transitive supply chain that k8s doesn't know about?

[mlieberman85]

To also provide some context. Whatever project(s) we end up working with as part of this proposal the idea would be to apply best practices and other guidance from CNCF as well as other community guidance to help secure supply chain. Examples include:

• CNCF Supply Chain Security Best Practices White Paper - https://github.com/cncf/tag-security/blob/main/supply-chain-security/supply-chain-security-paper/CNCF_SSCP_v1.pdf
• CNCF Secure Software Factory Reference Architecture - https://github.com/cncf/tag-security/blob/main/supply-chain-security/secure-software-factory/Secure_Software_Factory_Whitepaper.pdf
• OpenSSF SLSA - https://slsa.dev/ - both follow the practices and generate attestations
• Multiple other OpenSSF guidance and potentially tools.

[JonZeolla]

@mlieberman85 I'd like to contribute to this but the first supply chain security meeting I can join is 6/23. I will stay in touch via #tag-security-supply-chain-wg. Does it make sense to update the original post with the collaboration details yet?

[mlieberman85]

@mlieberman85 I'd like to contribute to this but the first supply chain security meeting I can join is 6/23. I will stay in touch via #tag-security-supply-chain-wg. Does it make sense to update the original post with the collaboration details yet?

I will edit. There will be a short discussion about the possibility of collaboration at the next sig-release meeting.

[mlieberman85]

These are k8s related threads that @justaugustus posted in kubernetes sig-release slack.

https://github.com/kubernetes/sig-release/blob/master/roadmap.md https://github.com/kubernetes/enhancements/issues/3027 https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/3027-slsa-compliance

These are related to roadmaps and work that kubernetes is working on for supply chain security. As an action I will be looking it over and providing feedback.

[anvega]

It's encouraging to hear conversations taking place between the workgroup crew and the different project stakeholders. One aspect of this proposed collaboration that isn't immediately clear is 'who 'is actually on the hook to carry out the work.

One motion here might be to incubate/facilitate discussions for supply chain security features/problems that don't have clear ownership.  The initial discovery can serve as a place to learn more so that we can shift towards ownership and execution. The actual delegation of responsibilities can from there be negotiated based the risk/threat level of the project, as well as on the maintainers' capacity and ability to take on the work.

As a baseline, we could provide the services, libraries, tools, and expertise to help other projects across the board secure their work. Following a 'paved road' model it would remain the responsibility of each project team to make use of these resources. In practice, this means that someone here would open up the issue and provide advice and coordination to help resolve the issue with a pull request. Staffing and operationalizing the day-to-day administration, as well as signing ceremonies, would be the responsibility of the project team.

As we continue to socialize proposals we should recognize that while the rationale as to why spend time and resources to harden the software supply chain are clear, asking already busy maintainers of high growth and high-velocity project to overhaul their CI/CD does represent significant overhead, introduces potentially disruptive change, and will take a toll on their work as the task will require their undivided attention for a period of time. Having a form of contract with clear expectations of who does the scoping, who takes on what, and for how long, will help ease acceptance.

To help inform what projects to prioritize we can build on top of the existing project maturity categorization to tier projects in the form of Risk/Threat Levels (RTL):

RTL-1 (Graduated): High-risk; very large deployment surface area; extremely high reputational impact RTL-2 (Incubated): High-risk; large-medium deployment surface area; high or potentially-high reputational impact RTL-3 (Sandbox): Medium-risk; medium deployment surface area and reputational impact

Projects identified as RTL-1 would see increased love and handholding compared to baseline help offered to RTL-2a projects. RTL-3 would get pointers to docs and white papers.

The last consideration here for now is that any CNCF project asking The Foundation for help with their CI/CD needs will be directed to use GitHub Actions as the preferred/recommended solution, not a "We'll help you host, set up, and manage Tekton or anything else you'd like". Worth looking at what the Project Services team (@jeefy and @amye ) is doing, and get engaged with them on why the policy is what it is. Maybe there is an opportunity to explore enablement for growing/evolving/sustaining this, but changing it requires changing it, not just going ahead. Doing our own shadow 'project service' is not even a distant second-best plan but a no-go.

[lumjjb]

We need to setup an in-person chat about this... I think we need to take a step back and understand what the motivations are from each side and what aligned goals there can be...

I want to just point out in this thread, since more people are getting tagged in... that the nature of the thread is discussion and not a commitment to anything yet.

[mlieberman85]

This makes sense. My intention for this proposal is to focus more on taking stuff and refining it, rather than wholesale building something new like a whole CI/CD pipeline for them. For example, cert-manager has some builds already and with a few tweaks could be following many more of the best practices with minimal additional overhead.

In the case of Kubernetes, it probably makes sense to just review what they've been doing already and see if there's any places we would suggest. From there I think we can figure out if there's a body of work, we (Security TAG) would be willing to commit to along with the Kubernetes SIGs.

[lumjjb]

We need to setup an in-person chat about this... I think we need to take a step back and understand what the motivations are from each side and what aligned goals there can be...

I will take this action item.

[anvega]

We need to setup an in-person chat about this... I think we need to take a step back and understand what the motivations are from each side and what aligned goals there can be...

I find the title of this issue to be descriptive enough. The goal is clear, the tactics not so much.

With a recurring meeting in place is probably a good use of time to make use of that existing space for this discussion. Not sure the chat warrants a separate meeting.

And just as important as the chat, we need to be able to turn around and capture down succinctly and with precision what it is that we are talking about doing here.

[lumjjb]

Actions items:

• Brainstorm in issue on what incentives there are
• Brainstorm some parts of the ref architecture and best practices
• Create a survey for how important is supply chain - what is the level of commitment to this if given some advice

[pxp928]

Actions items:

• Brainstorm in issue on what incentives there are
• Brainstorm some parts of the ref architecture and best practices
• Create a survey for how important is supply chain - what is the level of commitment to this if given some advice

I will take the action item to create a short survey that we can distribute to maintainers to determine how important is supply chain to them and their projects.

[pxp928]

CNCF Supply Chain Security Best Practices Survey to Maintainers

1. From a scale of 1 to 10, how important is software supply chain security to the projects you are maintaining? (SCALE: 1 to 10)
2. Are you currently actively looking to improve your supply chain security? (YES/NO)
3. Would you be interested in joining a pilot program for the CNCF Supply Chain Security WG to provide you recommendations and guidelines on how the project can align with SLSA? (YES/NO)
4. How many hours per week can you allocate to improving your project based on the recommendations and guidelines? (# of Hours)
5. What other actionable items would you like to see from the CNCF Supply Chain Security WG that would benefit your project with regards to supply chain security? (Free from answer)

[nadgowdas]

Anchore 2022 Software Supply Chain Security Report -- https://anchore.com/software-supply-chain-security-report-2022/

[lumjjb]

Notes from 06/09 meeting discussion

• Shouldn't be just SLSA - should point to CNCF best practices doc
• From a scale of 1-10 how secure do you think your software supply chain is? (maybe instead of a subjective question, have multiple questions to tease out how much of the best practices are being exercised) - @nadgowdas to take a stab at drafting this
  • What tools are you using for your supply chain? What are existing practices
• Have you been impacted by any type of supply chain incidents? (To what extent that this is not just a theoretical concern for the projects) - @apmarshall to take a stab at this.
  • Log4j
  • ...
• What does your stack look like?
  • languages (golang, ....)
  • CI (github actions, travis, circleCI, etc.)
  • Frameworks (docker, ...)
  • ...
• do you see supply chain as a producer or a consumer or both?
• What is the motivation/impetus around your consideration of supply chain security
  • Customer requirements
  • Compliance (company policy)
  • Compliance (regulatory)
  • Hype
  • Security is important to us

[apmarshall]

Draft of the "have you been impacted question":

"In the past year, how often have supply chain security incidents impacted your project/organization? Incidents may include disclosure of supply chain attacks/vulnerabilities in your dependencies, suspected or actual supply chain attacks on your project, or compromises of your organization resulting from a supply chain attack."

• Never
• Infrequently
• Occasionally
• Often

Draft of "do you see" question:

"When you refer to supply chain security within your organization, which of the following most closely matches what you mean:"

• The security of the dependencies/tools/platforms your organization uses or consumes
• The security of the artifacts or products your organization deploys/distributes
• Both are equally weighted in our supply chain considerations

[sudo-bmitch]

The first question covers frequency, but I'd want to include severity too. There's a big difference between "dependabot notifies me once a month to merge a PR" and "there was this one incident with solarwinds".

[apmarshall]

Agreed. How about:

What impact have supply chain incidents (including vulnerabilities or compromises of your dependencies, attacks on your development/build/delivery processes, or compromises of your org through supply chain attacks) had on your organization?

• Routine/Chore tasks (ie, dependency updates) only
• Low Impact
• Medium Impact
• High Impact
• Critical/Severe Impacts

Or some sort of scale to that effect, perhaps with the levels more clearly defined.

On Fri, Jun 10, 2022 at 21:13, Brandon Mitchell @.***> wrote:

The first question covers frequency, but I'd want to include severity too. There's a big difference between "dependabot notifies me once a month to merge a PR" and "there was this one incident with solarwinds".

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[nadgowdas]

Draft of "From a scale of 1-10 how secure do you think your software supply chain is? "

What are the most immediate concerns you want to address (say by end of year 2022)?

• SBOM Management (generation, storage, applications)
• CICD Pipeline Hardening
• Integrity assurance with signing and verification
• Provenance Collection
• Any other, please specify --

How much you rely on open-source tooling in securing your supply chain ?

• Use all open-source tooling
• Use all licensed or proprietary tooling
• Mix of (a) and (b) above (Can you quantify, e.g. 60/40?)

How much you are motivated to "start-from scratch"/modernize your entire supply chain pipeline (CICD, Build Infra, DevOps practices etc.)

• Very Much
• Can consider
• Not at all

[mlieberman85]

Based on the feedback and input from @apmarshall @pxp928, @nadgowdas, and @lumjjb to summarize I think this is the set of questions, will refine formatting and turn into an actual survey more as we go through:

For your project:

• What are most immediate concerns you want to address in software supply chain security?
• How much do you rely on external open-source tooling vs. licensed or proprietary tooling your supply chain?
• How motivated are you to secure your end to end supply chain pipeline and why?
• What impact have supply chain incidents (including vulnerabilities or compromises of your dependencies, attacks on your development/build/delivery processes, or compromises of your org through supply chain attacks) had on your organization or project?
• Has your org/project been impacted by a supply chain attack or compromise, to the best of your knowledge?
• When you refer to supply chain security within your organization or project, which of the following most closely matches what you mean
  • The security of the dependencies/tools/platforms your organization uses or consumes
  • The security of the artifacts or products your organization deploys/distributes
  • Both are equally weighted in our supply chain considerations
• Would you be interested in joining a pilot program for the CNCF Supply Chain Security WG to provide you recommendations and guidelines on how the project can align with?
• If yes, how many hours per week could your project dedicate to supply chain security engineering?
• What does your stack look like? e.g. languages, major frameworks, CI tools
• What is your primary motivator or securing your project/organization's supply chain?
• What else could the CNCF Security TAG be doing to help you secure your supply chain?

[lumjjb]

Survey questions link: https://docs.google.com/document/d/1hBlXif7F8Sf57YEubX_je5cMaEvg3RAYfvJcWlRpQqs/edit

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[mlieberman85]

Survey has gone out. Here is a copy of the announcement:

Security Enthusiasts! Supply chain attacks are on the rise. Open source maintainers must take action to mitigate the risk of compromise by commensurably securing the build infrastructure of their projects.

To help mitigate the impact of these attacks, the CNCF TAG Security group has created the CNCF Software Supply Chain Security Best Practices and the corresponding Secure Software Factory reference architecture. The goal of CNCF TAG Security's Supply Chain Security WG is to help open source organizations and projects evaluate gaps in their current software supply chain and provide a path forward to remediate based on the best practices and reference architecture.

We want you! This is where you (yes you!) come in. We're looking for CNCF project maintainers to sumit survey answers to this CNCF supply chain security survey, https://surveymonkey.com/r/CNCF-Supply-Chain-Security-Survey-2022, so that we can provide the cloud native community with data around the state of supply chain security within the CNCF itself! We envision this data being used to help the CNCF Security Tag in coming up with new projects and initiatives to help harden the supply chain for both CNCF projects as well as the open source community .

Ready to dive in?
If you are a maintainer of a CNCF project, answer this survey. If you have any questions, concerns, or feedback join the #tag-security-supply-chain-wg CNCF Slack channel!

References CNCF Supply chain security survey - https://surveymonkey.com/r/CNCF-Supply-Chain-Security-Survey-2022 CNCF Software Supply Chain Security Best Practices - https://github.com/cncf/tag-security/blob/main/supply-chain-security/supply-chain-security-paper/CNCF_SSCP_v1.pdf Secure Software Factory Refence Architecture - https://github.com/cncf/tag-security/blob/main/supply-chain-security/secure-software-factory/Secure_Software_Factory_Whitepaper.pdf Tag-security-supply-chain-wg Slack channel - https://cloud-native.slack.com/archives/C01KL0B4LKC

[stale[bot]]

This issue has been automatically marked as inactive because it has not had recent activity.

[anvega]

This proposal has been subsumed by workstreams in the Supply Chain WG.

[mnm678]

@eddie-knight this might relate to the security baseline work