Failed to start karydia container

[ionysos]

Description

If I try to delete or to perform a rolling update of karydia it fails with:

Error: failed to start container "pre-install-karydia": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "chdir to cwd (\"/root\") set in config.json failed: permission denied": unknown

Steps to reproduce

1. kubectl apply -f ./install/helm-service-account.yaml
2. helm init --service-account tiller
3. helm install ./install/charts --name karydia
4. kubectl delete pod $(kubectl get pods -n kube-system -l app=karydia -o jsonpath='{.items[0].metadata.name}') -n kube-system
5. Check karydia pod status with kubectl get pod $(kubectl get pods -n kube-system -l app=karydia -o jsonpath='{.items[0].metadata.name}') -n kube-system OR kubectl describe pod $(kubectl get pods -n kube-system -l app=karydia -o jsonpath='{.items[0].metadata.name}') -n kube-system

Expected behavior

This should work without any issues. The (new) karydia pod should get status Running, as well.

Environment

• Kubernetes version (use kubectl version): v1.15.2
• Cloud provider or hardware configuration: minikube
• [ ] Gardener cluster

[CodeClinch]

As discussed one solution could be that we annotate (annotation: don't use Karydia) pods that are already in the system.

[ionysos]

I provided a temporary workaround (#187) for this issue but we have to keep this in mind and, thus, have to clean these workaround things up with the final solution.

[CodeClinch]

Implementation idea: use webhook filtering by label and namespace

[ionysos]

As described at the Kubernetes (K8s) docs (v1.15; 2019/08/23): In addition to the matching request rules, it is now possible to add objectSelector and namespaceSelector blocks to our webhook configurations to gain an additional kind of filtering directly on webhook level which would decrease the load / requests to karydia.

But this approach does NOT exclude either objects or namespaces by name instead it excludes them via their labels. This means it is a very similar approach to our current exclusion handling via annotations but it is not on pod / container / karydia level but on webhook level.