Security of GitOps

[mateuszpruchniak]

Hi Folks

On the one hand, GitOps increase the security itself through limitation to a minimum of administrative access to infrastructure and configuration, eliminate human error, an audit of any change, declarative single source of truth, automatic verification, and policy enforcement. Using GitOps tools and technologies, organizations can mitigate the vectors of attack by reducing the number of people and machines that have access to the target system. On the other hand, more attention should be paid to securing Git repositories and CI/CD processes.

I would like to open a discussion about the security of GitOps and in particular what are the best security practices. And as a result create a dedicated document.

[chris-short]

I welcome this conversation. I would encourage linking to known good guides that already exist as appropriate. I'd also like to see suggestions of what we can do in the GitOps pipeline as well.

[mateuszpruchniak]

I will start :)

Basic security setup:

1. Add a policy to enforce signature verification to prevent from applying unauthorized changes on the cluster:
   • https://fluxcd.io/legacy/flux/references/git-gpg/
   • https://argo-cd.readthedocs.io/en/stable/user-guide/gpg-verification/
2. Enforce policy to block sensitive data being pushed to Git. In order to store secrets and credentials safely in a public or private Git repository, use Mozilla SOPS CLI or SealedSecrets to protect it.
   • https://github.com/mozilla/sops
   • https://github.com/bitnami-labs/sealed-secrets
3. Rotate SSH keys and Personal Access Tokens, guard against is unauthorized access to Git repositories
4. Enable two-factor authentication for all git users
5. Inclusion strategy in .gitignore file rather than exclusion
6. Hardening of GitOps process, e.g.:
   • https://github.blog/2020-07-20-hardening-your-github-enterprise-server/
7. Define access roles on a per-repository basis to ensure only users with valid access credentials are allowed to interact with the repository
8. Git hosting providers (such as GitHub) versus self-hosted Git servers
9. Monitoring of Git vulnerabilities, keeping Git and related tools up to date
10. Enforce policy, e.g.:
    • dedicated reviewers to accept Pull Requests, as the security review gate
    • use security scanner and analysis tool like Kubesec
    • verification of deployments, to prevent the deployment of pod with a critical vulnerability - tools like Grafeas, Kritis

of course, not to be missed: https://www.weave.works/blog/securing-gitops-pipelines

[chris-short]

Strong opinions, loosely held.

On point 1:
We might need to find a new Flux reference. In general, I think this is less important for some environments than others. I know gpg hinders adoption of some tooling too. I always caution involving gpg as a rule so this is me doing that. Yes, my commits are signed, but there are a lot of folks writing code today that don't understand the benefits of it. I'd say we need to point to something or explicitly state the benefits of signing commits.

On point 2:
I'd like to note there's some serious and substantial movement toward cosign and Chainguard's tooling. If there's a case we could highlight there, it'd be wise even if it was in its infancy, in my opinion given the reception Chainguard/cosign has been given.

On point 3:
This is good GitHub/GitLab hygiene but is often not done. I wonder if we should have a standards doc or at least point to one about proper care and feeding of GitHub and GitLab instances and not necessarily include specific things in our docs that are good hygiene. I don't want folks to only adopt what we say while missing other important things to do. See point 6.

On point 4:
See point 3.

On point 5:

Inclusion strategy in .gitignore file rather than exclusion

Never thought about it like that. Tell me MORE!

On point 6:
Perfect. We should point to this first maybe and link/highlight to what specifically you MUST do for GitOps security.

On point 9:
We're going to have to point to something on this one.

On point 10:
We should be pointing to one of our yet to be created docs for all this maybe?

[pjbgf]

Awesome point to raise @mateuszpruchniak!

GitOps bring some great practices that helps achieving a stronger security posture, but the strength of that posture largely depends on what it is built upon (i.e. people and processes).

When proposing a generic security best practices we also need to consider that some of those security requirements will be specific to the (security) posture a company wants or needs to uphold. Some companies have heavier compliance requirements (e.g. audits, regulations, etc) than others.

We may benefit by grouping some of the recommendations in themes. Then we can dive into a bit more detail when necessary, or simply add pointers to industry leading sources of such information (i.e. CIS for security recomendations for locking down Kubernetes).

For systems to be secure a holistic approach is generally necessary as the weakest link tend to be the entry-point for the average attacker. I would suggest we break down recommendations in some generic categories. And then add technology agnostic recommendations, which are then adapted for the specific scenario of each user. Here's a quick example of what I mean:

• Developer Security Posture
  • Basic security hygiene
  • Use of Secure keys or time-bound keys instead of long-lived keys.
  • When long-lived keys (SSH/GPG) are used, they must be protected by strong passwords.
  • From your list items: 3, 5 (like the idea, but not sure the cost/benefit)
  • Avoid using the development machine for other uses that may be riskier. Rely on Ephemeral VMs and containers to provide the correct level of isolation across different trust boundaries. (i.e. executing curl somenicetool.sh | bash in a machine that has long-lived SSH keys to production should be avoided)
• Source Storage Hardening (Git Repository, Bucket, etc)
  • Automatic CVE detection of the project's supply Chain (item 9).
  • Automatic detection of security issues in the project (SAST/DAST).
  • Only store sensitive information encrypted (item 2) and automate secret leakage detection.
  • Enforce least privileges based on Auth-Z/N.
  • Enforce peer reviews/segregation of duties.
  • From your list items: 3, 4, 6, 7, 8.
• Build and Continuous Integration Infrastructure
  • Use Ephemeral build runners to avoid cross build contamination.
  • Ensure the provenance of all tools and libraries on the "release path" of your applications. (i.e. avoid curl somenicetool.sh | bash)
  • All servers are locked-down following security best practices for the chosen distribution.
• Cluster GitOps Enforcement
  • Enforce:
  • Image Provenance of all internal and external components. (CIS benchmark)
  • Allow list of CVEs that can be executed in the cluster (i.e. unpatched/not critical)
  • Best practices for cluster configuration. (CIS benchmark)
  • Best practices for workload configuration. (CIS benchmark)

The above is far from being an exaustive list, but overall I think this could really benefit to focus on Security Principles, so it will be evergreen and applicable to any technology. Here's some of the principles (and examples) I think a secure GitOps implementation should observe:

• Segregation of duties
  • Repositories (or Buckets) cannot be modified unilaterally by a single user, some peer-review must be in place.
  • Bypass of peer-reviews or branch protections should be monitored.
• Least privilege
• Adoption of technology-specific Security Best Practices for all components involved (i.e. repository, CI/CD tools, cloud/on-prem).
  • Kubernetes and some Linux distros have CIS Benchmarks.
• Segregation/Isolation of Environments and Projects
  • Aligns with the least privilege concept, but extends it by ensuring the correct level of isolation across projects and environments. If one is compromised, others should not be affected.

I also think we should lean on the CNCF, they have some good material that I think we should point to:

• https://github.com/cncf/tag-security/blob/main/security-whitepaper/cloud-native-security-whitepaper.md
• https://github.com/cncf/tag-security/blob/main/supply-chain-security/supply-chain-security-paper/CNCF_SSCP_v1.pdf

Sorry for the long text. I tend to get carried away talking about security. :see_no_evil:

[mateuszpruchniak]

Awesome! Maybe we can start cooperating on a doc, what do you prefer HackMD or Google Docs?

[pjbgf]

Yep completely agree. I think Google docs would make it easier and accessible for most folks keen on collaborating.

[mateuszpruchniak]

@pjbgf @chris-short

I am not completely proficient in Google Docs, but I created docs based on the longest message :)

https://docs.google.com/document/d/1aqYabvs1-gurX1lYB5fzQDdptuA7j-LavGCSmxOFM_g

Please, ask for access and I will add anyone interested to contribute to the document.