[BUG] Remove hostPath volume mount from fluxconfig-agent

nathanweeks commented 6 months ago

Describe the bug The fluxconfig-agent controller uses a hostPath volume mount that attempts to access a /var/lib/docker/containers directory on the host. When the Kubernetes cluster pod hostPath volumes should only use allowed host paths policy is enforced (e.g., as part of the Kubernetes cluster pod security baseline standards for Linux-based workloads policy initiative) with "deny" effect, the fluxconfig-agent fails to start due to policy violation.

This hostPath volume mount seems to be cruft in fluxconfig-agent, since docker is no longer used as the container runtime for AKS.

To Reproduce

Run the following shell script:
```
#!/usr/bin/env sh
```

set -o errexit

readonly RESOURCE_GROUP=aks-rg readonly CLUSTER_NAME=aks-demo

az group create --resource-group ${RESOURCE_GROUP}

az policy assignment create \ --resource-group ${RESOURCE_GROUP} \ --policy '098fc59e-46c7-4d99-9b16-64990e543d75' \ --param '{"effect": { "value": "deny" }}'

az aks create \ -n ${CLUSTER_NAME} \ -g ${RESOURCE_GROUP} \ --node-vm-size Standard_B2s \ --node-osdisk-size 30 \ --node-count 1 \ --enable-addons azure-policy \ --tier free

az k8s-configuration flux create \ -g ${RESOURCE_GROUP} \ -c ${CLUSTER_NAME} \ -n cluster-config \ --namespace cluster-config \ -t managedClusters \ --scope cluster \ -u https://github.com/Azure/gitops-flux2-kustomize-helm-mt \ --branch main \ --kustomization name=infra path=./infrastructure prune=true


2. Check the events observed for the fluxconfig-agent replicaset:

$ az aks get-credentials -g aks-rg -n aks-demo ... $ kubectl -n flux-system get rs NAME DESIRED CURRENT READY AGE fluxconfig-agent-f594b6fdb 1 0 0 11m ... $ kubectl -n flux-system events rs/fluxconfig-agent-f594b6fdb ... 76s (x9 over 12m) Warning FailedCreate ReplicaSet/fluxconfig-agent-f594b6fdb (combined from similar events): Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev3hostfilesystem-5f58ca0751a242787f0c] HostPath volume {"hostPath": {"path": "/var/lib/docker/containers", "type": ""}, "name": "varlibdockercontainers"} is not allowed, pod: fluxconfig-agent-f594b6fdb-hxbdb. Allowed path: []



**Expected behavior**
The fluxconfig-agent should start successfully when the previously-mentioned policy is enabled & enforced with "deny" effect.

**Screenshots**
See above

**Environment (please complete the following information):**
Azure Cloud Shell, with the following versions:
 -  CLI Version 2.58.0
 - Kubernetes version v1.29.2

**Additional context**
This issue was previously reported in https://github.com/Azure/azure-policy/issues/1161 . The suggestion to modify the policy to exclude the `flux-system` namespace is a reasonable workaround, though addressing the root cause by removing the inapplicable hostPath volume mount from fluxconfig-agent would seem to be an appropriate long-term solution.

CC: @haithamshahin333