Closed ludydoo closed 3 years ago
Nice to see distroless in the wild. You cannot kill sidecars with either the k8sapi or kubelet executors. It is not possible to fix. Instead, use PNS, Docker or Emissary executors.
Hi @alexec
The /quitquitquit
endpoint works well. Though, it introduces additional delay for each task.
Perhaps a possible improvement would be that
func (c *k8sAPIClient) KillContainer(pod *corev1.Pod, container *corev1.ContainerStatus, sig syscall.Signal) error {
Could poll the sidecar container status to see if it was already killed, instead of waiting for the command to return an error. But that's probably not a priority
sidecars:
- image: pause
name: pause
volumeMounts:
- mountPath: /bin
name: busybox
volumes:
- hostPath:
path: /opt/busybox
type: DirectoryOrCreate
name: busybox
volumes:
- name: busybox
emptyDir: {}
initContainers:
- image: busybox
name: busybox
command:
- sh
- -c
- |
cp /bin/busybox /opt/busybox/
cd /opt/busybox
busybox --list | xargs -I{} ln -s busybox {}
volumeMounts:
- mountPath: /opt/busybox
name: busybox
distroless container security
kill $(pgrep <sidecar command>)
slow
I think we need something better here.
Can you confirm if ISTIO contains a /bin/kill
file?
This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.
Summary
Using istio-9.1-distroless with automatic sidecar injection. The workflow executes the main container, then tries to kill the sidecar container using exec.
https://github.com/argoproj/argo-workflows/blob/e6fa41a1b91be2e56884ca16427aaaae4558fa00/workflow/executor/k8sapi/client.go#L99
Since the istio-proxy container does not have a
sh
shell, there is a timeout error.Workaround
Add a
curl
sidecar which will trigger the/quitquitquit
istio sidecar endpoint.As a bonus, this is the script I use to wait for istio proxy. Sometimes either curl/wget are not available, so using netcat:
This is the kill sidecar using netcat (busybox wget cannot POST):
Workaround drawbacks
Seems that argo
wait
container does not recognize that the sidecar was killed by/quitquitquit
, and waits for the timeout anyways, which adds additional delays.Diagnostics
What Kubernetes provider are you using?
1.20.2
What version of Argo Workflows are you running?
2.12.9
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.