GoogleCloudPlatform / alloydb-auth-proxy

A utility for connecting securely to your AlloyDB instances
https://cloud.google.com/alloydb/docs/auth-proxy/overview?hl=hu
Apache License 2.0
59 stars 11 forks source link

using alloydb-auth-proxy as k8s sidecar is not terminating with exit code 0 #727

Open strimpak opened 9 hours ago

strimpak commented 9 hours ago

Bug Description

This happens in lots of cases, but the easiest way to reproduce this is through a k8s job:

apiVersion: batch/v1
kind: Job
metadata:
  name: test-alloydb-auth-proxy
  namespace: default
spec:
  template:
    metadata:
      labels:
        app: test-alloydb-auth-proxy
    spec:
      containers:
      - name: test-alloydb-auth-proxy
        image: hello-world:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
      initContainers:
      - name: alloydb-auth-proxy
        image: gcr.io/alloydb-connectors/alloydb-auth-proxy:1.11.2-bullseye
        imagePullPolicy: IfNotPresent
        restartPolicy: Always
        args:
        - --structured-logs
        - --prometheus
        - --address=0.0.0.0
        - --max-sigterm-delay=30s
        - --http-port=9130
        - --health-check
        - --http-address=0.0.0.0
        # connection string
        - projects/<project>/locations/<location>/clusters/<cluster-id>/instances/<instance-id>
        resources:
          limits:
            memory: 64Mi
          requests:
            cpu: 50m
            memory: 64Mi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
        startupProbe:
          httpGet:
            path: /startup
            port: 9130
          periodSeconds: 1
          timeoutSeconds: 5
          failureThreshold: 20
        livenessProbe:
          httpGet:
            path: /liveness
            port: 9130
          initialDelaySeconds: 10
          periodSeconds: 60
          timeoutSeconds: 30
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 9130
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 6
      restartPolicy: Never
      serviceAccountName: testsa
      terminationGracePeriodSeconds: 30
  backoffLimit: 4

status of pod from job:

  containerStatuses:
  - containerID: containerd://05dd9afe90387d7ee9e2f4f1987fd62bdd5b69ed759cca57edaf899979e87d63
    image: docker.io/library/hello-world:latest
    imageID: docker.io/library/hello-world@sha256:305243c734571da2d100c8c8b3c3167a098cab6049c9a5b066b6021a60fcb966
    lastState: {}
    name: test-alloydb-auth-proxy
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://05dd9afe90387d7ee9e2f4f1987fd62bdd5b69ed759cca57edaf899979e87d63
        exitCode: 0
        finishedAt: "2024-11-28T17:00:39Z"
        reason: Completed
        startedAt: "2024-11-28T17:00:39Z"
  initContainerStatuses:
  - containerID: containerd://9b3721ce694ae4328cdebbaf3624ac27ee1c7457c6756f76915b89f91228ec13
    image: gcr.io/alloydb-connectors/alloydb-auth-proxy:1.11.2-bullseye
    imageID: gcr.io/alloydb-connectors/alloydb-auth-proxy@sha256:8aa6a3dd67efc929923eeec5f5057898f1a4bf7767a05bd36639b4c3d7a321dc
    lastState: {}
    name: alloydb-auth-proxy
    ready: false
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://9b3721ce694ae4328cdebbaf3624ac27ee1c7457c6756f76915b89f91228ec13
        exitCode: 143
        finishedAt: "2024-11-28T17:00:40Z"
        reason: Error
        startedAt: "2024-11-28T17:00:36Z"

This seems that sigterm received by container that it terminates is not handled correctly. This is not happening with cloudsql proxy.

Example code (or command)

// paste your code here

Stacktrace

logs from stern:

~ stern test-alloydb-auth-proxy
+ test-alloydb-auth-proxy-qn58r › alloydb-auth-proxy
test-alloydb-auth-proxy-qn58r alloydb-auth-proxy {"severity":"INFO","timestamp":"2024-11-28T17:00:36.761Z","message":"Authorizing with Application Default Credentials"}
test-alloydb-auth-proxy-qn58r alloydb-auth-proxy {"severity":"INFO","timestamp":"2024-11-28T17:00:37.377Z","message":"[xyz.us-central1.xyz.xyz-default] Listening on [::]:5432"}
test-alloydb-auth-proxy-qn58r alloydb-auth-proxy {"severity":"INFO","timestamp":"2024-11-28T17:00:37.378Z","message":"The proxy has started successfully and is ready for new connections!"}
test-alloydb-auth-proxy-qn58r alloydb-auth-proxy {"severity":"INFO","timestamp":"2024-11-28T17:00:37.378Z","message":"Starting health check server at 0.0.0.0:9130"}
test-alloydb-auth-proxy-qn58r alloydb-auth-proxy {"severity":"INFO","timestamp":"2024-11-28T17:00:40.279Z","message":"SIGTERM signal received. Shutting down..."}
+ test-alloydb-auth-proxy-qn58r › test-alloydb-auth-proxy
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy Hello from Docker!
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy This message shows that your installation appears to be working correctly.
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy To generate this message, Docker took the following steps:
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy  1. The Docker client contacted the Docker daemon.
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy  2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy     (amd64)
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy  3. The Docker daemon created a new container from that image which runs the
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy     executable that produces the output you are currently reading.
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy  4. The Docker daemon streamed that output to the Docker client, which sent it
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy     to your terminal.
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy To try something more ambitious, you can run an Ubuntu container with:
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy  $ docker run -it ubuntu bash
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy Share images, automate workflows, and more with a free Docker ID:
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy  https://hub.docker.com/
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy For more examples and ideas, visit:
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy  https://docs.docker.com/get-started/
test-alloydb-auth-proxy-qn58r test-alloydb-auth-proxy
- test-alloydb-auth-proxy-qn58r › test-alloydb-auth-proxy
- test-alloydb-auth-proxy-qn58r › alloydb-auth-proxy


### Steps to reproduce?

1. provided in the description of the bug
2. ?
3. ?
...

### Environment

1. OS type and version: debian & default docker image
2. AlloyDB Proxy version: 1.11.2 & 1.11.3 reproduced
3. Proxy invocation command (for example, `./alloydb-auth-proxy --port 6000 <INSTANCE_URI>`):

### Additional Details

_No response_
strimpak commented 8 hours ago

I've seen that there is a flag added in 1.11.3 --exit-zero-sigterm that fixes this. Maybe you can adjust docs/examples to reflect it.