search

hashicorp / vault-k8s

First-class support for Vault and Kubernetes.
Mozilla Public License 2.0
785 stars 170 forks source link

agent-inject-command break command after any bash operators @ vault 1.70 #246

Open mattgialelis opened 3 years ago

mattgialelis commented 3 years ago

Description As of vault 1.7.0 there is an issue with the agent-inject-command not fully reading in bash commands after any operators. e.g ; , && , ||, '>'

Multiple different methods to escape the operators was tried none seemed to make any difference the command on its own is:

"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'"

Which is used to restart the deployment on a secret change.

This causes the commands either to not run at all or just error out like the logs provided below, ive tested with vault 1.5.4 and 1.6.3 which both work well and breaks on 1.7.0

Vault 1.7.0 (Broken)

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.7.0
2021-04-14T08:48:04.821Z [INFO]  sink.file: creating file sink
2021-04-14T08:48:04.821Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
             Version Sha: 4e222b85c40a810b74400ee3c54449479e32bb9f

 (Removed extra lines due to spam)

[DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":""},"Templates":[{"Backup":false,"Command":"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'","CommandTimeout":30000000000,"Contents":"{{- with secret \"secret/myapp/config\" -}}\n  vars:\n  {{ range $k, $v := .Data }}\n    {{ $k }}: {{ $v }}\n  {{ end }}\n{{- end }}\n{{- with secret \"secret/generic/config\" -}}\n  {{ range $k, $v := .Data }}\n    {{ $k }}: {{ $v }}\n  {{ end }}\n{{- end }}\n","CreateDestDirs":true,"Destination":"/vault/secrets/config.yml","ErrMissingKey":false,"Exec":{"Command":"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'","Enabled":true,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://10.200.0.150:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}

[DEBUG] (runner) vault.read(secret/myapp/config) is still needed
[DEBUG] (runner) vault.read(secret/generic/config) is still needed
[INFO] (runner) executing command "sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'" from "(dynamic)" => "/vault/secrets/config.yml"
[INFO] (child) spawning: sh -c { if [ -f /vault/secrets/config.check ]
if: line 1: syntax error: unexpected end of file (expecting "}")
[INFO] (runner) stopping
[DEBUG] (runner) stopping watcher
[DEBUG] (watcher) stopping all views

Vault 1.6.3 (working)

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.6.3
             Version Sha: b540be4b7ec48d0dd7512c8d8df9399d6bf84d76

 (Removed extra lines due to spam)

2021/04/14 08:27:57.112387 [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":""},"Templates":[{"Backup":false,"Command":"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'","CommandTimeout":30000000000,"Contents":"{{- with secret \"secret/myapp/config\" -}}\n  vars:\n  {{ range $k, $v := .Data }}\n    {{ $k }}: {{ $v }}\n  {{ end }}\n{{- end }}\n{{- with secret \"secret/generic/config\" -}}\n  {{ range $k, $v := .Data }}\n    {{ $k }}: {{ $v }}\n  {{ end }}\n{{- end }}\n","CreateDestDirs":true,"Destination":"/vault/secrets/config.yml","ErrMissingKey":false,"Exec":{"Command":"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'","Enabled":true,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://10.200.0.150:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}

2021/04/14 08:27:57.281910 [DEBUG] (runner) vault.read(secret/myapp/config) is still needed
2021/04/14 08:27:57.281916 [DEBUG] (runner) vault.read(secret/generic/config) is still needed
2021/04/14 08:27:57.281930 [INFO] (runner) executing command "sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'" from "(dynamic)" => "/vault/secrets/config.yml"
2021/04/14 08:27:57.282119 [INFO] (child) spawning: sh -c { if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }
2021/04/14 08:27:57.283618 [DEBUG] (runner) watching 2 dependencies
2021/04/14 08:27:57.283735 [DEBUG] (runner) all templates rendered
2021/04/14 08:27:57.283633 [INFO] (runner) stopping

To Reproduce Steps to reproduce the behavior:

  1. Setup Vault-Kube auth
  2. Deploy application using provided manifest
  3. See error (vault-agent-init container) kubectl logs POD-NAME vault-agent-init

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-agent-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-example
  template:
    metadata:
      labels:
        app: app-example
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-secret-config.yml: "secret/myapp/config"
        vault.hashicorp.com/log-level: debug
        vault.hashicorp.com/agent-inject-template-config.yml: |
          {{- with secret "secret/myapp/config" -}}
            vars:
            {{ range $k, $v := .Data }}
              {{ $k }}: {{ $v }}
            {{ end }}
          {{- end }}
        vault.hashicorp.com/role: 'example'
        vault.hashicorp.com/agent-inject-command-config.yml: "sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'"
    spec:
      containers:
        - image: alpine:latest
          name: alpine
          command:
            - /bin/sh
            - "-c"
            - "sleep 60m"
          imagePullPolicy: IfNotPresent
      serviceAccountName: vault-auth

kubectl logs POD-NAME vault-agent-init ( this is where all logs provided are from )

Expected behavior Command to be fully executed as it was in previous version of vaults-agent

Environment

Additional context We are using an external vault server and the agent injector only from the Vault Helm Chart 

    image: hashicorp/vault-k8s:0.9.0
    imageID: docker-pullable://hashicorp/vault-k8s@sha256:65731b0513c95f683ee52528e6ccf24f6de0092700e869cdc5ff5d8354b5d86e

Currently the only workaround is to make our own vault docker image with a bash script which we can call as a single line command store in /usr/local/bin/reload-pods reload-pods deployment-name

Which does work as it does not require any extra operators in the annotations provided to vault

Have also just tested using the configmap method with the same results

   "auto_auth" = {
      "method" = {
        "config" = {
          "role" = "example"
        }
        "type" = "kubernetes"
      }

      "sink" = {
        "config" = {
          path = "/home/vault/.vault-token"
        }

        "type" = "file"
      }
    }

    "exit_after_auth" = true
    "pid_file" = "/home/vault/.pid"

    "template" = {
      "contents" = "{{- with secret \"secret/myapp/config\" -}}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}{{ end }}{{- end }}"
      "destination" = "/vault/secrets/config"
      "command" =  "sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'"
    }
    "vault" = {
      "address" = "http://10.200.0.150:8200"
    }
jasonodonnell commented 3 years ago

His @mattgialelis, Vault Agent uses consul-template under the hood so I would suggest filing an issue there. Likely a dependency was updated and is causing issues now.

pjastrzabek commented 2 years ago

Just hit that. Trying to restart thanos

vault.hashicorp.com/agent-inject-command-thanos-store: "ps aux | grep -v grep | grep thanos && killall -9 thanos"

but only ps is being called :/

2021-11-11T12:29:15.644Z [INFO] (runner) executing command "ps aux | grep -v grep | grep thanos && killall -9 thanos" from "(dynamic)" => "/vault/secrets/thanos-store"
2021-11-11T12:29:15.644Z [INFO] (child) spawning: ps aux

That forces me to use extra wrapper scripts and mount them into containers within pod :(