Validate configuration changes

[ameukam]

if possible, any configuration change introducted should be validated with presubmits prowjobs.

There are different and tools we can explore :

• Globally, define policies with rego language and enforce them with Open Policy Agent
• units tests and e2e tests with terratest
  • Doesn't cover Kubernetes resources
• Kubernetes resources validation with conftest (also based on rego)

/priority backlog

[ameukam]

cc @spiffxp

[spiffxp]

Was literally just about to file this issue, thank you!

/remove-priority backlog /priority important-longterm /sig testing /area access /area cluster-mgmt /area cluster-infra

[spiffxp]

/area prow

[spiffxp]

Presubmits we have thus far:

• sig-release:
  • if /k8s.gcr.io/**/{images,promoter-manifest}.yaml changes
  • pull-k8sio-cip: dry-run container image promoter
  • pull-k8sio-cip-vuln: (for vuln-check-test branch) dry-run container image promoter with vuln-severity-threshold=1
  • if /infra/gcp/backup_tools changes
  • run infra/gcp/backup_tools/backup_test
• sig-contribex:
  • if /groups changes
  • pull-k8sio-groups-test: run cd groups && go test (some policy enforcement)
• sig-testing:
  • pull-k8sio-yamllint: verifies all yaml in this repo is valid yaml (could enforce style conventions)

[spiffxp]

• Globally, define policies with rego language and enforce them with Open Policy Agent
• Kubernetes resources validation with conftest (also based on rego)

Really, really good idea. Way more interested in conftest as a starting point, seems like basically any YAML is fair game.

I am pretty interested in how much this could allow us to share enforcement logic at different phases in our deployment lifecycle, eg: presubmit, admission, etc.

[ameukam]

Also since 0.13, Terraform has embebbed mechanism for variable validation : https://www.hashicorp.com/blog/custom-variable-validation-in-terraform-0-13.

[ameukam]

/milestone v1.22

[spiffxp]

An update on where we're at.

We now have a container gcr.io/k8s-staging-infra-tools/k8s-infra:latest which is used by all of our CI jobs (ref: https://github.com/kubernetes/k8s.io/pull/2134 and https://github.com/kubernetes/test-infra/pull/22463). This image is capable of running tools like contest, opa and terraform

We use this image to run a few sets of tests:

• hack/verify-executable.sh - verifies **/*.sh and **/*.py files in the repo are executable (except for lib*.sh files which are intended to be sourced, not executed) (ref: https://github.com/kubernetes/k8s.io/pull/2204)
• hack/verify-shellcheck.sh - runs shellcheck for **/*.sh files in the repo (ref: https://github.com/kubernetes/k8s.io/pull/2201)
• hack/verify-terraform.sh - runs terraform init; terraform validate for a given cluster in infra/gcp/clusters (ref: https://github.com/kubernetes/k8s.io/pull/1732)
• make -C groups test - runs go test to validate a few hardcoded polices for groups/**/*.yaml (ref: https://github.com/kubernetes/k8s.io/pull/399)

We've taken tentative steps toward extracting our configuration data into YAML (ref: https://github.com/kubernetes/k8s.io/pull/2188). We should spend some time deciding what we want this to look like.

Some ideas:

• continue down the path of a custom home-grown schema
  • continue extracting hardcoded data out of infra/gcp/*.sh (lists of services, lists of buckets, shared constants, etc.)
  • shard infra.yaml into multiple yaml files
  • write non-bash code (e.g. go or python) to encode our schema
  • validate our schema (using aforementioned code, or conftest/opa)
• example off-the-shelf tooling for configuration-as-code, e.g.
  • lean into terraform, use yaml as data source to drive modules (ref: https://github.com/kubernetes/k8s.io/issues/516)
  • lean into crossplane (ref: https://github.com/kubernetes/k8s.io/issues/1885)
  • lean into cloud asset inventory (ref: https://github.com/kubernetes/k8s.io/issues/1981)

[ameukam]

example off-the-shelf tooling for configuration-as-code, e.g.

Also investigate https://cloud.google.com/config-connector/docs/overview

[spiffxp]

/milestone v1.23

[ameukam]

/milestone v1.24 /help wanted

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale /lifecycle frozen /milestone clear