Secrets are not mounted in main container, when using vault-agent-init

sando38 commented 11 months ago

Preflight Checklist

[X] I have searched the issue tracker for an issue that matches the one I want to file, without success.
[X] I am not looking for support or already pursued the available support channels without success.
[X] I agree to follow the Code of Conduct.

Operator Version

1.21.2

Installation Type

Official Helm chart

Bank-Vaults Version

No response

Kubernetes Version

1.27.2

Kubernetes Distribution/Provisioner

Scaleway

Expected Behavior

The main container has secrets available at e.g. /vault/secrets

Actual Behavior

No volumeMount for /vault/secrets is available in the main containers (my application, see config example below).

Steps To Reproduce

This does not work:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fpush-config-apns-vault-agent
  namespace: fpush
  labels:
    app.kubernetes.io/name: fpush
    my-app.kubernetes.io/name: fpush
data:
  config.hcl: |
    vault {
    }
    auto_auth {
      method "kubernetes" {
        mount_path = "auth/kubernetes"
        config = {
          role = "fpush"
        }
      }
      sink "file" {
        config = {
          path = "/vault/.vault-token"
        }
      }
    }
    template {
    contents = <<EOH
    {{- with secret "/secret/applications/production/fpush_apns" -}}
    {{ base64Decode .Data.data.certFile }}
    {{- end }}
    EOH
      destination = "/vault/secrets/apns_certificates.p12"
      exec {
        command = ["true"]
      }
    }
    template {
    contents = <<EOH
    {{- with secret "/secret/applications/production/ejabberd_listener_fpush" }}
    {
    "component": {
        "componentHostname": "fpush-apns.example.net",
        "componentKey": "{{ .Data.data.password_apns }}",
        "serverHostname": "ejabberd.ejabberd.svc.cluster.local",
        "serverPort": 5347
    },
    {{ end }}
    {{- with secret "/secret/applications/production/fpush_apns" }}
    "pushModules": {
        "iOS": {
            "type": "apple",
            "is_default_module": true,
            "apns": {
                "certFilePath": "/vault/secrets/apns_certificates.p12",
                "certPassword": "{{ .Data.data.certPassword }}",
                "environment": "{{ .Data.data.environment }}",
                "topic": "{{ .Data.data.topic }}"
            },
            "ratelimit": {
                "ratelimitTime": "20s",
                "hardRatelimitTime": "20s",
                "ratelimitCleanupInterval": "300s",
                "enabled": false
            }
        }
    },
    "timeout": {
        "xmppconnectionError": "20s"
    }
    }
    {{ end }}
    EOH
      destination = "/vault/secrets/settings.json"
      exec {
        command = ["true "]
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  namespace: fpush
  labels:
    app: busybox
spec:
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
        sidecar.istio.io/inject: "true"
      annotations:
        traffic.sidecar.istio.io/excludeOutboundPorts: "8200"
        vault.security.banzaicloud.io/vault-addr: "https://vault.vault:8200"
        vault.security.banzaicloud.io/vault-role: "fpush"
        vault.security.banzaicloud.io/vault-skip-verify: "false"
        vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
        vault.security.banzaicloud.io/vault-path: "kubernetes"
        vault.security.banzaicloud.io/vault-agent: "true"
        vault.security.banzaicloud.io/vault-agent-init: "true"
        vault.security.banzaicloud.io/vault-agent-configmap: "fpush-config-apns-vault-agent"
        vault.security.banzaicloud.io/vault-agent-once: "true"
    spec:
      securityContext:
        fsGroup: 1000
      serviceAccountName: default
      containers:
      - name: busybox
        image: busybox
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsUser: 65532
          runAsGroup: 1000
          runAsNonRoot: true
          privileged: false
          capabilities:
            drop: [ALL]
        args: ["/bin/sh", "-c", "sleep 120"]

However, when I trick the environment by also using vault-env to inject a pseudo environment variable, a volumeMount /vault is created which makes the secrets available from the vault-agent init container.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fpush-config-apns-vault-agent
  namespace: fpush
  labels:
    app.kubernetes.io/name: fpush
    my-app.kubernetes.io/name: fpush
data:
  config.hcl: |
    vault {
    }
    auto_auth {
      method "kubernetes" {
        mount_path = "auth/kubernetes"
        config = {
          role = "fpush"
        }
      }
      sink "file" {
        config = {
          path = "/vault/.vault-token"
        }
      }
    }
    template {
    contents = <<EOH
    {{- with secret "/secret/applications/production/fpush_apns" -}}
    {{ base64Decode .Data.data.certFile }}
    {{- end }}
    EOH
      destination = "/vault/secrets/apns_certificates.p12"
      exec {
        command = ["true"]
      }
    }
    template {
    contents = <<EOH
    {{- with secret "/secret/applications/production/ejabberd_listener_fpush" }}
    {
    "component": {
        "componentHostname": "fpush-apns.example.net",
        "componentKey": "{{ .Data.data.password_apns }}",
        "serverHostname": "ejabberd.ejabberd.svc.cluster.local",
        "serverPort": 5347
    },
    {{ end }}
    {{- with secret "/secret/applications/production/fpush_apns" }}
    "pushModules": {
        "iOS": {
            "type": "apple",
            "is_default_module": true,
            "apns": {
                "certFilePath": "/vault/secrets/apns_certificates.p12",
                "certPassword": "{{ .Data.data.certPassword }}",
                "environment": "{{ .Data.data.environment }}",
                "topic": "{{ .Data.data.topic }}"
            },
            "ratelimit": {
                "ratelimitTime": "20s",
                "hardRatelimitTime": "20s",
                "ratelimitCleanupInterval": "300s",
                "enabled": false
            }
        }
    },
    "timeout": {
        "xmppconnectionError": "20s"
    }
    }
    {{ end }}
    EOH
      destination = "/vault/secrets/settings.json"
      exec {
        command = ["true"]
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  namespace: fpush
  labels:
    app: busybox
spec:
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
        sidecar.istio.io/inject: "true"
      annotations:
        traffic.sidecar.istio.io/excludeOutboundPorts: "8200"
        vault.security.banzaicloud.io/vault-addr: "https://vault.vault:8200"
        vault.security.banzaicloud.io/vault-role: "fpush"
        vault.security.banzaicloud.io/vault-skip-verify: "false"
        vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
        vault.security.banzaicloud.io/vault-path: "kubernetes"
        vault.security.banzaicloud.io/vault-agent: "true"
        vault.security.banzaicloud.io/vault-agent-init: "true"
        vault.security.banzaicloud.io/vault-agent-configmap: "fpush-config-apns-vault-agent"
        vault.security.banzaicloud.io/vault-agent-once: "true"
    spec:
      securityContext:
        fsGroup: 1000
      serviceAccountName: default
      containers:
      - name: busybox
        image: busybox
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsUser: 65532
          runAsGroup: 1000
          runAsNonRoot: true
          privileged: false
          capabilities:
            drop: [ALL]
        args: ["/bin/sh", "-c", "sleep 120"]
        env:
          - name: PUSH_ENVIRONMENT
            value: "${vault:secret/data/applications/production/fpush_apns#environment}"

Configuration

No response

Logs

No response

Additional Information

No response

akijakya commented 10 months ago

Hi @sando38, I just tried out your first example, and it worked for me when I removed the vault and auto_auth parts from the template (in this case I believe the data for connecting to Vault is used from the annotations). If you try it again and still doesn't work, I'd like to ask you to provide some more info or logs from the failing container! And thanks for using Bank-Vaults! 🙂

sando38 commented 10 months ago

Interesting. I will look into it again and provide feedback.

I know, the question is unrelated, but, if I use consul templates with the following annotations in my statefulset:

spec:
  template:
    metadata:
      annotations:
        vault.security.banzaicloud.io/vault-addr: "https://vault.vault:8200"
        vault.security.banzaicloud.io/vault-role: "ejabberd"
        vault.security.banzaicloud.io/vault-skip-verify: "false"
        vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
        vault.security.banzaicloud.io/vault-path: "kubernetes"
        vault.security.banzaicloud.io/vault-agent: "true"
        vault.security.banzaicloud.io/vault-ct-configmap: "ejabberd-config-vault-agent"
        vault.security.banzaicloud.io/vault-ct-once: "false"
        vault.security.banzaicloud.io/vault-ct-secrets-mount-path: "/vault/secrets"

The vault-agent sidecar consul-template does not renew its token and eventually it can not update the secrets anymore. This is the consul template I use:


data:
  config.hcl: |
    vault {
      vault_agent_token_file = "/vault/.vault-token"
      retry {
        backoff = "1s"
      }
    }
    template {
    contents = <<EOH
    ...

I have one hour ttl in the kubernetes auth, but not maxTTL defined, meaning it should be able to renew for 32 days.

Any idea? Thanks in advance!

sando38 commented 10 months ago

Okay, when I remove the vault and auto_auth from my configs above, I get the following error from the vault-agent container:

error validating configuration: no auto_auth, cache, or listener block found in config

sando38 commented 10 months ago

By the way: I use vault version 1.15.0

github-actions[bot] commented 8 months ago

Thank you for your contribution! This issue has been automatically marked as stale because it has no recent activity in the last 60 days. It will be closed in 20 days, if no further activity occurs. If this issue is still relevant, please leave a comment to let us know, and the stale label will be automatically removed.

ramizpolic commented 8 months ago

Have you had any luck on this @sando38? I am going to look into this this week so let me know if this is still relevant.

sando38 commented 7 months ago

I am not sure if it is solved, but let's close it. I will come back to you once I dived into the configs again.

bank-vaults / vault-secrets-webhook