Create release Compliant Kubernetes Kubespray 2.26.0

Overview

[!note] Whenever you need to change access from operator admin to admin@example.com prefer to re-login by clearing the ~/.kube/cache/oidc-login cache instead of impersonation --as=admin@example.com.

Pre-QA steps
Install QA steps
Upgrade QA steps
Post-QA steps
Release steps

# Pre-QA steps

[x] Complete the feature freeze step
[x] Complete the staging step
[ ] Complete all pre-QA steps in the internal checklist

# Install QA steps

Kubespray install scenario

Infrastructure provider

[ ] Elastx
[x] Safespring
[ ] UpCloud

Configuration

[x] Flavor - Prod
[x] Dex IdP - Google
[x] Dex Static User - Enabled and admin@example.com added as an application developer

Commands
```bash # configure yq4 -i '.grafana.user.oidc.allowedDomains += ["example.com"]' "${CK8S_CONFIG_PATH}/sc-config.yaml" yq4 -i 'with(.opensearch.extraRoleMappings[]; with(select(.mapping_name != "all_access"); .definition.users += ["admin@example.com"]))' "${CK8S_CONFIG_PATH}/sc-config.yaml" yq4 -i '.user.adminUsers += ["admin@example.com"]' "${CK8S_CONFIG_PATH}/wc-config.yaml" yq4 -i '.dex.enableStaticLogin = true' "${CK8S_CONFIG_PATH}/sc-config.yaml" pushd ~/path/to/apps/ # apply ./bin/ck8s apply sc ./bin/ck8s apply wc popd ```
[x] Set the environment variable NAMESPACE to an application developer namespace (this cannot be a subnamespace)
[x] Set the environment variable DOMAIN to the environment domain

Automated tests

[!note] As platform administrator

[x] Successful ./bin/ck8s test sc|wc
[x] From tests/ successful make build-main
[ ] From tests/ successful make ctr-run-end-to-end

Kubernetes access

[!note] As platform administrator

[x] Can login as platform administrator via Dex with IdP

[!note] As application developer admin@example.com

[x] Can login as application developer admin@example.com via Dex with static user

[x] Can list access

kubectl -n "${NAMESPACE}" auth can-i --list

[x] Can delegate admin access

$ kubectl -n "${NAMESPACE}" edit rolebinding extra-workload-admins
# Add some subject
subjects:
  # You can specify more than one "subject"
  - kind: User
    name: jane # "name" is case sensitive
    apiGroup: rbac.authorization.k8s.io

[x] Can delegate view access

$ kubectl edit clusterrolebinding extra-user-view
# Add some subject
subjects:
  # You can specify more than one "subject"
  - kind: User
    name: jane # "name" is case sensitive
    apiGroup: rbac.authorization.k8s.io

[x] Cannot run with root by default

kubectl apply -n "${NAMESPACE}" -f - <<EOF
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-root-nginx
spec:
podSelector:
  matchLabels:
    app: root-nginx
policyTypes:
  - Ingress
  - Egress
ingress:
  - {}
egress:
  - {}
---
apiVersion: v1
kind: Pod
metadata:
labels:
  app: root-nginx
name: root-nginx
spec:
restartPolicy: Never
containers:
  - name: nginx
    image: nginx:stable
    resources:
      requests:
        memory: 64Mi
        cpu: 250m
      limits:
        memory: 128Mi
        cpu: 500m
EOF

Hierarchical Namespaces

[!note] As application developer admin@example.com

[x] Can create a subnamespace by following the application developer docs

Commands
```bash kubectl apply -n "${NAMESPACE}" -f - <
[x] Ensure the default roles, rolebindings, and networkpolicies propagated

Commands
```bash kubectl get role,rolebinding,netpol -n "${NAMESPACE}" kubectl get role,rolebinding,netpol -n "${NAMESPACE}-qa-test" ```

Harbor

[!note] As application developer admin@example.com

[x] Can login as application developer via Dex with static user

Steps
- Login to Harbor with `admin@example.com` ```bash xdg-open "https://harbor.${DOMAIN}" ``` - Login to Harbor with the admin user and promote `admin@example.com` to admin - Re-login with `admin@example.com`
[x] Can create projects and push images by following the application developer docs
[x] Can configure image pull secret by following the application developer docs
[x] Can scan image for vulnerabilities
[x] Configure project to disallow vulnerabilities
- Try to pull image with vulnerabilities, should fail
```
docker pull "harbor.${DOMAIN}/${REGISTRY_PROJECT}/ck8s-user-demo:${TAG}"
```
[x] Configure project to allow vulnerabilities
- Try to pull image with vulnerabilities, should succeed
```
docker pull "harbor.${DOMAIN}/${REGISTRY_PROJECT}/ck8s-user-demo:${TAG}"
```

Gatekeeper

[!note] As application developer admin@example.com

[x] Can list OPA rules
```
kubectl get constraints
```

[!note] Using the user demo helm chart

Set NAMESPACE to an application developer namespaces Set PUBLIC_DOCS_PATH to the path of the public docs repo

[x] With invalid image repository, try to deploy, should warn due to constraint

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/ck8s-user-demo" \
  --set image.repository="${REGISTRY_PROJECT}/ck8s-user-demo" \
  --set image.tag="${TAG}" \
  --set ingress.hostname="demoapp.${DOMAIN}"

[x] With invalid image tag, try to deploy, should fail due to constraint

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/ck8s-user-demo" \
  --set image.repository="harbor.${DOMAIN}/${REGISTRY_PROJECT}/ck8s-user-demo" \
  --set image.tag=latest \
  --set ingress.hostname="demoapp.${DOMAIN}"

[x] With unset networkpolicies, try to deploy, should warn due to constraint

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/ck8s-user-demo" \
  --set image.repository="harbor.${DOMAIN}/${REGISTRY_PROJECT}/ck8s-user-demo" \
  --set image.tag="${TAG}" \
  --set ingress.hostname="demoapp.${DOMAIN}" \
  --set networkPolicy.enabled=false

[x] With unset resources, try to deploy, should fail due to constraint

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/ck8s-user-demo" \
  --set image.repository="harbor.${DOMAIN}/${REGISTRY_PROJECT}/ck8s-user-demo" \
  --set image.tag="${TAG}" \
  --set ingress.hostname="demoapp.${DOMAIN}" \
  --set resources.requests=null

[x] With valid values, try to deploy, should succeed

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/ck8s-user-demo" \
  --set image.repository="harbor.${DOMAIN}/${REGISTRY_PROJECT}/ck8s-user-demo" \
  --set image.tag="${TAG}" \
  --set ingress.hostname="demoapp.${DOMAIN}"

cert-manager and ingress-nginx

[!note] As platform administrator

[x] All certificates ready including user demo
[x] All ingresses ready including user demo
- [x] Endpoints are reachable
- [x] Status includes correct IP addresses

Metrics

[!note] As platform administrator

[x] Can login to platform administrator Grafana via Dex with IdP
[x] Dashboards are available and viewable
[x] Metrics are available from all clusters

[!note] As application developer admin@example.com

[x] Can login to application developer Grafana via Dex with static user

Steps
- Login to Grafana with `admin@example.com` ```bash xdg-open "https://grafana.${DOMAIN}" ``` - Login to Grafana with the admin user and promote `admin@example.com` to admin - Re-login with `admin@example.com`
[x] Welcome dashboard presented first
[x] Dashboards are available and viewable
[x] Metrics are available from all clusters
[x] Metrics are available from user demo application
[x] CISO dashboards available and working

List
- [Backup / Backup Status](https://elastisys.io/compliantkubernetes/ciso-guide/backup/) - [Cryptography / NGINX Ingress Controller](https://elastisys.io/compliantkubernetes/ciso-guide/cryptography/) - [Intrusion Detection / Falco](https://elastisys.io/compliantkubernetes/ciso-guide/intrusion-detection/) - [Policy-as-Code / Gatekeeper](https://elastisys.io/compliantkubernetes/ciso-guide/policy-as-code/) - [Network Security / NetworkPolicy](https://elastisys.io/compliantkubernetes/ciso-guide/network-security/) - [Capacity Management / Kubernetes Cluster Status](https://elastisys.io/compliantkubernetes/ciso-guide/capacity-management/) - [Vulnerability / Trivy Operator Dashboard](https://elastisys.io/compliantkubernetes/ciso-guide/vulnerability/)

Alerts

[!note] As platform administrator

[x] No alert open except Watchdog, CPUThrottlingHigh and FalcoAlert
- Can be seen in the alert section in platform administrator Grafana

[!note] As application developer admin@example.com

[x] Access Prometheus following the application developer docs
[x] Prometheus picked up user demo ServiceMonitor and PrometheusRule
[x] Access Alertmanager following the application developer docs
[x] Alertmanager Watchdog firing

Logs

[!note] As platform administrator

[x] Can login to OpenSearch Dashboards via Dex with IdP
[x] Indices created (authlog, kubeaudit, kubernetes, other)
[x] Indices managed (authlog, kubeaudit, kubernetes, other)
[x] Logs available (authlog, kubeaudit, kubernetes, other)
[x] Snapshots configured

[!note] As application developer admin@example.com

[x] Can login to OpenSearch Dashboards via Dex with static user
[x] Welcome dashboard presented first
[x] Logs available (kubeaudit, kubernetes)
[x] CISO dashboards available and working

Falco

[!note] As platform administrator

[x] Deploy the falcosecurity/event-generator to generate events in wc

Commands
```bash # Install kubectl create namespace event-generator kubectl label namespace event-generator owner=operator helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update helm -n event-generator install event-generator falcosecurity/event-generator \ --set securityContext.runAsNonRoot=true \ --set securityContext.runAsGroup=65534 \ --set securityContext.runAsUser=65534 \ --set podSecurityContext.fsGroup=65534 \ --set config.actions="" # Uninstall helm -n event-generator uninstall event-generator kubectl delete namespace event-generator ```
[x] Logs are available in OpenSearch Dashboards
[x] Logs are relevant

Network policies

[x] No dropped packets in NetworkPolicy Grafana dashboard

Infrastructure tests

[x] Able to run terraform plan without changes
[x] Able to add nodes without issues
[x] Able to remove nodes without issues

# Upgrade QA steps

Kubespray upgrade scenario

[!note] The upgrade is done as part of the checklist.

Infrastructure provider

[ ] Elastx
[ ] Safespring
[x] UpCloud

Configuration

[x] Flavor - Prod
[x] Dex IdP - Google
[x] Dex Static User - Enabled and admin@example.com added as an application developer

Commands
```bash # configure yq4 -i '.grafana.user.oidc.allowedDomains += ["example.com"]' "${CK8S_CONFIG_PATH}/sc-config.yaml" yq4 -i 'with(.opensearch.extraRoleMappings[]; with(select(.mapping_name != "all_access"); .definition.users += ["admin@example.com"]))' "${CK8S_CONFIG_PATH}/sc-config.yaml" yq4 -i '.user.adminUsers += ["admin@example.com"]' "${CK8S_CONFIG_PATH}/wc-config.yaml" yq4 -i '.dex.enableStaticLogin = true' "${CK8S_CONFIG_PATH}/sc-config.yaml" pushd ~/path/to/apps/ # apply ./bin/ck8s apply sc ./bin/ck8s apply wc popd ```
[x] Set the environment variable NAMESPACE to an application developer namespace (this cannot be a subnamespace)
[x] Set the environment variable DOMAIN to the environment domain

Upgrade

[x] Can upgrade according to the migration docs for this version

Automated tests

[!note] As platform administrator

[x] Successful ./bin/ck8s test sc|wc
[ ] From tests/ successful make build-main
[ ] From tests/ successful make ctr-run-end-to-end

Kubernetes access

[!note] As platform administrator

[x] Can login as platform administrator via Dex with IdP

[!note] As application developer admin@example.com

[x] Can login as application developer admin@example.com via Dex with static user

[x] Can list access

kubectl -n "${NAMESPACE}" auth can-i --list

[x] Can delegate admin access

$ kubectl -n "${NAMESPACE}" edit rolebinding extra-workload-admins
# Add some subject
subjects:
  # You can specify more than one "subject"
  - kind: User
    name: jane # "name" is case sensitive
    apiGroup: rbac.authorization.k8s.io

[x] Can delegate view access

$ kubectl edit clusterrolebinding extra-user-view
# Add some subject
subjects:
  # You can specify more than one "subject"
  - kind: User
    name: jane # "name" is case sensitive
    apiGroup: rbac.authorization.k8s.io

[x] Cannot run with root by default

kubectl apply -n "${NAMESPACE}" -f - <<EOF
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-root-nginx
spec:
podSelector:
  matchLabels:
    app: root-nginx
policyTypes:
  - Ingress
  - Egress
ingress:
  - {}
egress:
  - {}
---
apiVersion: v1
kind: Pod
metadata:
labels:
  app: root-nginx
name: root-nginx
spec:
restartPolicy: Never
containers:
  - name: nginx
    image: nginx:stable
    resources:
      requests:
        memory: 64Mi
        cpu: 250m
      limits:
        memory: 128Mi
        cpu: 500m
EOF

Hierarchical Namespaces

[!note] As application developer admin@example.com

[x] Can create a subnamespace by following the application developer docs

Commands
```bash kubectl apply -n "${NAMESPACE}" -f - <
[x] Ensure the default roles, rolebindings, and networkpolicies propagated

Commands
```bash kubectl get role,rolebinding,netpol -n "${NAMESPACE}" kubectl get role,rolebinding,netpol -n "${NAMESPACE}-qa-test" ```

Harbor

[!note] As application developer admin@example.com

[x] Can login as application developer via Dex with static user

Steps
- Login to Harbor with `admin@example.com` ```bash xdg-open "https://harbor.${DOMAIN}" ``` - Login to Harbor with the admin user and promote `admin@example.com` to admin - Re-login with `admin@example.com`
[x] Can create projects and push images by following the application developer docs
[x] Can configure image pull secret by following the application developer docs
[x] Can scan image for vulnerabilities
[x] Configure project to disallow vulnerabilities
- Try to pull image with vulnerabilities, should fail
```
docker pull "harbor.${DOMAIN}/${REGISTRY_PROJECT}/welkin-user-demo:${TAG}"
```
[x] Configure project to allow vulnerabilities
- Try to pull image with vulnerabilities, should succeed
```
docker pull "harbor.${DOMAIN}/${REGISTRY_PROJECT}/welkin-user-demo:${TAG}"
```

Gatekeeper

[!note] As application developer admin@example.com

[x] Can list OPA rules
```
kubectl get constraints
```

[!note] Using the user demo helm chart

Set NAMESPACE to an application developer namespaces Set PUBLIC_DOCS_PATH to the path of the public docs repo

[x] With invalid image repository, try to deploy, should warn due to constraint

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/welkin-user-demo" \
  --set image.repository="${REGISTRY_PROJECT}/welkin-user-demo" \
  --set image.tag="${TAG}" \
  --set ingress.hostname="demoapp.${DOMAIN}"

[x] With invalid image tag, try to deploy, should fail due to constraint

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/welkin-user-demo" \
  --set image.repository="harbor.${DOMAIN}/${REGISTRY_PROJECT}/welkin-user-demo" \
  --set image.tag=latest \
  --set ingress.hostname="demoapp.${DOMAIN}"

[x] With unset networkpolicies, try to deploy, should warn due to constraint

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/welkin-user-demo" \
  --set image.repository="harbor.${DOMAIN}/${REGISTRY_PROJECT}/welkin-user-demo" \
  --set image.tag="${TAG}" \
  --set ingress.hostname="demoapp.${DOMAIN}" \
  --set networkPolicy.enabled=false

[x] With unset resources, try to deploy, should fail due to constraint

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/welkin-user-demo" \
  --set image.repository="harbor.${DOMAIN}/${REGISTRY_PROJECT}/welkin-user-demo" \
  --set image.tag="${TAG}" \
  --set ingress.hostname="demoapp.${DOMAIN}" \
  --set resources.requests=null

[x] With valid values, try to deploy, should succeed

helm -n "${NAMESPACE}" upgrade --atomic --install demo "${PUBLIC_DOCS_PATH}/user-demo/deploy/welkin-user-demo" \
  --set image.repository="harbor.${DOMAIN}/${REGISTRY_PROJECT}/welkin-user-demo" \
  --set image.tag="${TAG}" \
  --set ingress.hostname="demoapp.${DOMAIN}"

cert-manager and ingress-nginx

[!note] As platform administrator

[x] All certificates ready including user demo
[x] All ingresses ready including user demo
- [x] Endpoints are reachable
- [x] Status includes correct IP addresses

Metrics

[!note] As platform administrator

[x] Can login to platform administrator Grafana via Dex with IdP
[x] Dashboards are available and viewable
[x] Metrics are available from all clusters

[!note] As application developer admin@example.com

[x] Can login to application developer Grafana via Dex with static user

Steps
- Login to Grafana with `admin@example.com` ```bash xdg-open "https://grafana.${DOMAIN}" ``` - Login to Grafana with the admin user and promote `admin@example.com` to admin - Re-login with `admin@example.com`
[x] Welcome dashboard presented first
[x] Dashboards are available and viewable
[x] Metrics are available from all clusters
[x] Metrics are available from user demo application
[x] CISO dashboards available and working

List
- [Backup / Backup Status](https://elastisys.io/compliantkubernetes/ciso-guide/backup/) - [Cryptography / NGINX Ingress Controller](https://elastisys.io/compliantkubernetes/ciso-guide/cryptography/) - [Intrusion Detection / Falco](https://elastisys.io/compliantkubernetes/ciso-guide/intrusion-detection/) - [Policy-as-Code / Gatekeeper](https://elastisys.io/compliantkubernetes/ciso-guide/policy-as-code/) - [Network Security / NetworkPolicy](https://elastisys.io/compliantkubernetes/ciso-guide/network-security/) - [Capacity Management / Kubernetes Cluster Status](https://elastisys.io/compliantkubernetes/ciso-guide/capacity-management/) - [Vulnerability / Trivy Operator Dashboard](https://elastisys.io/compliantkubernetes/ciso-guide/vulnerability/)

Alerts

[!note] As platform administrator

[x] No alert open except Watchdog, CPUThrottlingHigh and FalcoAlert
- Can be seen in the alert section in platform administrator Grafana

[!note] As application developer admin@example.com

[x] Access Prometheus following the application developer docs
[x] Prometheus picked up user demo ServiceMonitor and PrometheusRule
[x] Access Alertmanager following the application developer docs
[x] Alertmanager Watchdog firing

Logs

[!note] As platform administrator

[x] Can login to OpenSearch Dashboards via Dex with IdP
[x] Indices created (authlog, kubeaudit, kubernetes, other)
[x] Indices managed (authlog, kubeaudit, kubernetes, other)
[x] Logs available (authlog, kubeaudit, kubernetes, other)
[x] Snapshots configured

[!note] As application developer admin@example.com

[x] Can login to OpenSearch Dashboards via Dex with static user
[x] Welcome dashboard presented first
[x] Logs available (kubeaudit, kubernetes)
[x] CISO dashboards available and working

Falco

[!note] As platform administrator

[x] Deploy the falcosecurity/event-generator to generate events in wc

Commands
```bash # Install kubectl create namespace event-generator kubectl label namespace event-generator owner=operator helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update helm -n event-generator install event-generator falcosecurity/event-generator \ --set securityContext.runAsNonRoot=true \ --set securityContext.runAsGroup=65534 \ --set securityContext.runAsUser=65534 \ --set podSecurityContext.fsGroup=65534 \ --set config.actions="" # Uninstall helm -n event-generator uninstall event-generator kubectl delete namespace event-generator ```
[ ] Logs are available in OpenSearch Dashboards
[ ] Logs are relevant

Network policies

[x] No dropped packets in NetworkPolicy Grafana dashboard

Infrastructure tests

[x] Able to run terraform plan without changes
[x] Able to add nodes without issues
[x] Able to remove nodes without issues

# Post-QA steps

[x] Complete the code freeze step
[x] Complete all post-QA steps in the internal checklist

# Release steps

[x] Complete the release step
[x] Complete the update public release notes step
[x] Complete the update the main branch step
[x] Complete all post release steps in the internal checklist

elastisys / compliantkubernetes-kubespray