estafette / estafette-gke-preemptible-killer

Kubernetes controller to spread preemption for preemtible VMs in GKE to avoid mass deletion after 24 hours
https://helm.estafette.io
MIT License
448 stars 72 forks source link

Add the ability to annotate pods. #89

Closed lucazz closed 7 months ago

lucazz commented 3 years ago

First of all, I'd like to say, this is an AWESOME project and thanks for the great work that was put into it so far!

Now to this PR's changes description and the problem it tries to solve.

We're running istio in our GKE clusters and if we allow istio's sidecars to be injected on this project's pods, it actually prevents some API calls from being made, preventing this project from actually doing what it is supposed to do:

❯❯❯ kubectl describe sa preemptible-killer
Name:                preemptible-killer
Namespace:           foo
Labels:              app.kubernetes.io/instance=preemptible-killer
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=preemptible-killer
                     app.kubernetes.io/version=1.2.6
                     helm.sh/chart=estafette-gke-preemptible-killer-1.2.6
Annotations:         iam.gke.io/gcp-service-account: gke-preemptible-killer@foo-project.iam.gserviceaccount.com
                     meta.helm.sh/release-name: preemptible-killer
                     meta.helm.sh/release-namespace: foo
Image pull secrets:  <none>
Mountable secrets:   preemptible-killer-token-4pdz7
Tokens:              preemptible-killer-token-4pdz7
Events:              <none>
 ❯❯❯ gcloud --project=foo iam service-accounts get-iam-policy "gke-preemptible-killer@foo-project.iam.gserviceaccount.com"
bindings:
- members:
  - serviceAccount:foo-project.svc.id.goog[foo/preemptible-killer]
  role: roles/iam.workloadIdentityUser
etag: <redacted>=
version: 1
Updates are available for some Cloud SDK components.  To install them,
please run:
  $ gcloud components update
 ❯❯❯ stern preemptible-killer -c estafette-gke-preemptible-killer
[...]
preemptible-killer-89866db6f-z4z75 estafette-gke-preemptible-killer {"level":"error","error":"Get https://10.1.128.1:443/api/v1/nodes?labelSelector=cloud.google.com%2Fgke-preemptible%3Dtrue: dial tcp 10.1.128.1:443: connect: connection refused","time":"2021-02-09T20:29:49Z","message":"Error while getting the list of preemptible nodes"}
[...]

Dropping in a pod annotation like this to prevent these pods to have envoy (Istio's sidecar) in it fixes this issue:

sidecar.istio.io/inject: "false"

So this PR aims to give users the ability to annotate their pods with that option or any other annotation that might make sense to their deployments.

Let me know if there's anything else you'd need to change in order to get this merged.