search

spiffe / spiffe-helper

The SPIFFE Helper is a tool that can be used to retrieve and manage SVIDs on behalf of a workload
Apache License 2.0
43 stars 40 forks source link

Kubernetes Sidecar Usecase #115

Open kfox1111 opened 8 months ago

kfox1111 commented 8 months ago

It would be great if spiffe-helper could be used as a sidecar under Kubernetes.

This would require two different modes of operation to function well.

  1. a new flag for running in job mode. This would block until it could fetch the cert/key/ca, and then exit 0 on completion. This would run as a k8s initContainer and ensure initial cert/key/ca creation before the workload starts.
  2. a new flag specifying a pid file. Instead of running a command and then signaling the process it forked, it would just notify the pid that preexists as specified in the file. This would enable it to signal a process in a different container, in the same pod.

A container image would also be needed. Requested here: https://github.com/spiffe/spiffe-helper/issues/107

kfox1111 commented 8 months ago

Maybe for a separate issue... To support the new k8s sidecar support in 1.28+, also a health check would be needed. https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/

nstott commented 8 months ago
  1. would definitely be cleaner. definitely a nice to have. we have spiffe-helper running in a sidecar now, and the primary container will generally restart once or twice while it waits for the helper to fetch the svids.

for 2, we'd still have to share the pid namespace though. I don't think I understand that usecase

kfox1111 commented 8 months ago

@nstott I ported the mysql bits to work with the bitnami mysql helm chart here: https://github.com/spiffe/helm-charts-hardened/blob/mysql-using-spire/examples/mysql-using-spire/mysql-values.yaml

The problem is, in order to trigger a reload of the certs in mysql, you need a viable mysql client. So you need spire-helper available in the mysql container. I did mange to do some twisty bits with init containers to piece it together at runtime. But its kind of ugly.

Instead, if you had shared pids enabled and pid signaling, the sidecar could just contain spiffe-helper and send a signal over to a container sidecar that has the mysql client ready to go to adapt the pid signal to a mysql reload certs sql command.

I've prototyped that using the pid pr and a prototype CSI driver for spiffe-helper to make it possible to do this kind of thing: https://github.com/spiffe/helm-charts-hardened/pull/166/files#diff-95b3020faa9ac7ddba1d84fc7dcf64f0965725c3f3a3d71be001a5d2847ba4c8

That example actually works and signals nginx to properly reload when the certs get updated.

keeganwitt commented 1 month ago

A usecase I'm now looking at is if you have restartPolicy: Always on your pod, and are pre-1.28 Kubernetes, then you need a way to have the sidecar container exit when the main container exits. This is kinda inverting the direction of what's discussed here. Should this need be a separate ticket?

kfox1111 commented 1 month ago

I thought a restartPolicy was ignored when its in a terminating state?

But, if thats not the case, maybe a preStop hook would help

keeganwitt commented 1 month ago

I think I mis-stated the scenario. The restartPolicy would be either Never or OnFailure in this case. And let's say the pod is part of a job, so the main container runs and exist normally. But the SPIFFE Helper sidecar is still running. "Terminating" isn't a pod phase. In this case, the pod phase would be "Running" because at least one container (the helper) is still running, the main container status would be "Terminated" and the sidecar container would have a status of "Running".

What we'd want in this scenario is after the main container exits successfully, the sidecar knows it needs to exit too.

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-states