[vault-secrets-webhook] Failed to create pod

Hi, I'm getting this issue, and I can see the following error from the replicaset:

Error creating: Internal error occurred: failed calling admission webhook "pods.vault-secrets-webhook.admission.banzaicloud.com": an error on the server ("{\"response\":{\"uid\":\"c55bece2-9738-11e9-a521-028284ad0f6c\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \\"$KUBERNETES_SECRET_NAME\\" not found\"}}}") has prevented the request from succeeding

Hi @misanche thanks for the issue. From this error message I would say the required Kubernetes secret is not found on the cluster/namespace where the pod wants to run.

Hi @baluchicken, thank you for your rapid response, It was working previously but this is how I install it:

kubectl create namespace vault-wh
kubectl label ns vault-wh name=vault-wh
helm upgrade --namespace vault-wh --install vault-wh banzaicloud-stable/vault-secrets-webhook --tls

Logs from the webhook pod:

time="2019-06-25T11:13:44Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:45Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:45Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:47Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:49Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:54Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:14:05Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:14:25Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:14:55Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:15:06Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:16:28Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:17:38Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:19:12Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:23:06Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:24:40Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found"

@misanche can you please check if you are using the latest version of the chart?

Sure @sagikazarmark,

from the deployment I can see the following annotation: "chart": "vault-secrets-webhook-0.3.17"

@misanche can you please check your pod specs which will be mutated by this webhook? Is there any pod which contains more than one ImagePullSecret?

@baluchicken yes it has.

I have removed now, the wh annotations and envs.


{
  "kind": "Deployment",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "message-service-v1",
    "namespace": "mesh",
    "selfLink": "/apis/extensions/v1beta1/namespaces/mesh/deployments/message-service-v1",
    "uid": "c43f2b95-919b-11e9-a521-028284ad0f6c",
    "resourceVersion": "12755080",
    "generation": 30,
    "creationTimestamp": "2019-06-18T07:36:20Z",
    "labels": {
      "app": "message-service",
      "version": "v1"
    },
    "annotations": {
      "deployment.kubernetes.io/revision": "30",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"message-service\",\"version\":\"v1\"},\"name\":\"message-service-v1\",\"namespace\":\"mesh\"},\"spec\":{\"replicas\":1,\"template\":{\"metadata\":{\"labels\":{\"app\":\"message-service\",\"version\":\"v1\"}},\"spec\":{\"containers\":[{\"image\":\"271960289458.dkr.ecr.us-east-2.amazonaws.com/reach-istio:message_5\",\"imagePullPolicy\":\"Always\",\"name\":\"message-service\",\"ports\":[{\"containerPort\":8080}]}],\"imagePullSecrets\":[{\"name\":\"$KUBERNETES_SECRET_NAME\"}]}}}}\n"
    }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "app": "message-service",
        "version": "v1"
      }
    },
    "template": {
      "metadata": {
        "creationTimestamp": null,
        "labels": {
          "app": "message-service",
          "version": "v1"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "message-service",
            "image": "<imageUrl>",
            "ports": [
              {
                "containerPort": 8080,
                "protocol": "TCP"
              }
            ],
            "env": [
              {
                "name": "JAEGER_SERVICE_NAME",
                "value": "message-service"
              },
              {
                "name": "JAEGER_ENDPOINT",
                "value": "http://jaeger-collector.istio-system.svc:14268/api/traces"
              },
              {
                "name": "JAEGER_PROPAGATION",
                "value": "b3"
              },
              {
                "name": "JAEGER_SAMPLER_TYPE",
                "value": "const"
              },
              {
                "name": "JAEGER_SAMPLER_PARAM",
                "value": "1"
              },
              {
                "name": "TEST",
                "value": "2"
              }
            ],
            "resources": {},
            "terminationMessagePath": "/dev/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "Always"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {},
        "imagePullSecrets": [
          {
            "name": "$KUBERNETES_SECRET_NAME"
          }
        ],
        "schedulerName": "default-scheduler"
      }
    },
    "strategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "maxUnavailable": 1,
        "maxSurge": 1
      }
    },
    "revisionHistoryLimit": 10,
    "progressDeadlineSeconds": 2147483647
  },
  "status": {
    "observedGeneration": 30,
    "replicas": 1,
    "updatedReplicas": 1,
    "readyReplicas": 1,
    "availableReplicas": 1,
    "conditions": [
      {
        "type": "Available",
        "status": "True",
        "lastUpdateTime": "2019-06-18T07:36:21Z",
        "lastTransitionTime": "2019-06-18T07:36:21Z",
        "reason": "MinimumReplicasAvailable",
        "message": "Deployment has minimum availability."
      }
    ]
  }
}

If I remove the imagePullSecrets, I see the following error:

Error creating: Internal error occurred: failed calling admission webhook "pods.vault-secrets-webhook.admission.banzaicloud.com": an error on the server ("{\"response\":{\"uid\":\"16a60567-9749-11e9-a521-028284ad0f6c\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"cannot create client for registry: Get /v2/: unsupported protocol scheme \\\"\\\"\"}}}") has prevented the request from succeeding

I managed to find your problem. We just added a new feature to the webhook which kick in when there is no container commands and container args specified. https://github.com/banzaicloud/bank-vaults/blob/master/docs/mutating-webhook/README.md#using-charts-without-explicit-containercommand-and-containerargs

@baluchicken sorry, I don't have it very clear, what could I do now?

Regards

@misanche Is the "$KUBERNETES_SECRET_NAME" Kubernetes secret available in the pods namespace?

@baluchicken sec, it's not mine, I'm trying to deploy it without imagePullSecret or request more information.

thank you for your support

@baluchicken I have tried to remove and still doesn't work, but If I remove the imagePullSecrets and all the annotations and env vars for vault webhook, I'm able to deploy it.

{ "kind": "Pod", "apiVersion": "v1", "metadata": { "name": "message-service-v1-649ddcff9-6vjdq", "generateName": "message-service-v1-649ddcff9-", "namespace": "default", "selfLink": "/api/v1/namespaces/default/pods/message-service-v1-649ddcff9-6vjdq", "uid": "1fbe95f6-974d-11e9-a521-028284ad0f6c", "resourceVersion": "12770126", "creationTimestamp": "2019-06-25T13:28:31Z", "labels": { "app": "message-service", "pod-template-hash": "205887995", "version": "v1" }, "annotations": { "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container message-service" }, "ownerReferences": [ { "apiVersion": "apps/v1", "kind": "ReplicaSet", "name": "message-service-v1-649ddcff9", "uid": "1fbcd8b8-974d-11e9-a521-028284ad0f6c", "controller": true, "blockOwnerDeletion": true } ] }, "spec": { "volumes": [ { "name": "default-token-qv2hq", "secret": { "secretName": "default-token-qv2hq", "defaultMode": 420 } } ], "containers": [ { "name": "message-service", "image": "<image>", "ports": [ { "containerPort": 8080, "protocol": "TCP" } ], "env": [ { "name": "JAEGER_SERVICE_NAME", "value": "message-service" }, { "name": "JAEGER_ENDPOINT", "value": "http://jaeger-collector.istio-system.svc:14268/api/traces" }, { "name": "JAEGER_PROPAGATION", "value": "b3" }, { "name": "JAEGER_SAMPLER_TYPE", "value": "const" }, { "name": "JAEGER_SAMPLER_PARAM", "value": "1" } ], "resources": { "requests": { "cpu": "100m" } }, "volumeMounts": [ { "name": "default-token-qv2hq", "readOnly": true, "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount" } ], "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "imagePullPolicy": "Always" } ], "restartPolicy": "Always", "terminationGracePeriodSeconds": 30, "dnsPolicy": "ClusterFirst", "serviceAccountName": "default", "serviceAccount": "default", "nodeName": "ip-10-50-85-254.us-east-2.compute.internal", "securityContext": {}, "schedulerName": "default-scheduler", "tolerations": [ { "key": "node.kubernetes.io/not-ready", "operator": "Exists", "effect": "NoExecute", "tolerationSeconds": 300 }, { "key": "node.kubernetes.io/unreachable", "operator": "Exists", "effect": "NoExecute", "tolerationSeconds": 300 } ], "priority": 0 }, "status": { "phase": "Running", "conditions": [ { "type": "Initialized", "status": "True", "lastProbeTime": null, "lastTransitionTime": "2019-06-25T13:28:31Z" }, { "type": "Ready", "status": "True", "lastProbeTime": null, "lastTransitionTime": "2019-06-25T13:28:49Z" }, { "type": "ContainersReady", "status": "True", "lastProbeTime": null, "lastTransitionTime": null }, { "type": "PodScheduled", "status": "True", "lastProbeTime": null, "lastTransitionTime": "2019-06-25T13:28:31Z" } ], "hostIP": "10.50.85.254", "podIP": "100.96.2.140", "startTime": "2019-06-25T13:28:31Z", "containerStatuses": [ { "name": "message-service", "state": { "running": { "startedAt": "2019-06-25T13:28:49Z" } }, "lastState": {}, "ready": true, "restartCount": 0, "image": "<image>", "imageID": "<imageId>", "containerID": "docker://3e29851e1350c31086fcdb4afcb15b50cdcf7c78d5927c9ec682f941ff127d34" } ], "qosClass": "Burstable" } }

I have removed imagePullSecrets because they are not needed, I get this error now:

2019/06/25 13:28:29 registry.ping url=/v2/
time="2019-06-25T13:28:29Z" level=error msg="admission webhook error: cannot create client for registry: Get /v2/: unsupported protocol scheme \"\""

I guess you are using a private repository for your images. If yes you have two options:

supply the access secret to it on your deployment using the imagePullSecrets field, and be sure that secret is accessible from the same namespace
supply the command for your container using the command field, so the webhook will not try to download and determine the image entrypoint/command for you.

Hi @baluchicken I tried to follow your feedback and now I can see how the pod, but is stuck at Waiting: PodInitializing,

In the vault-agent init container I see this error:

2019-06-25T14:44:10.208Z [ERROR] auth.handler: error authenticating: error="Error making API request.
 URL: PUT https://vault.vault:8200/v1/auth/kubernetes/login
Code: 400. Errors:
 * missing client token" backoff=1.360566553
2019-06-25T14:44:11.569Z [INFO]  auth.handler: authenticating

@baluchicken I don't know if this is the problem but I install the operator and the webhook in this way: `helm repo add banzaicloud-stable http://kubernetes-charts.banzaicloud.com/branch/master

helm repo update

helm install banzaicloud-stable/vault-operator --namespace=vault --tls --name=vault-operator

helm upgrade --install vault-operator banzaicloud-stable/vault-operator --set=etcd-operator.enabled=true --set=etcd-operator.etcdOperator.commandArgs.cluster-wide=true --namespace=vault --tls

kubectl apply -f rbac.yaml -n vault

kubectl apply -f cr-etcd-ha.yaml -n vault

kubectl create namespace vault-wh

kubectl label ns vault-wh name=vault-wh

helm upgrade --namespace vault-wh --install vault-wh banzaicloud-stable/vault-secrets-webhook --tls

rbac.yaml:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: vault
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: vault-secrets
rules:
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - "*"
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: vault-secrets
roleRef:
  kind: Role
  name: vault-secrets
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: vault

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault
    namespace: vault

cr-etcd-ha.yaml:

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
spec:
  size: 2
  image: vault:1.1.0
  bankVaultsImage: banzaicloud/bank-vaults:latest

  # Specify the ServiceAccount where the Vault Pod and the Bank-Vaults configurer/unsealer is running
  serviceAccount: vault

  # Specify the Service's type where the Vault Service is exposed
  serviceType: LoadBalancer

  # Specify how many nodes you would like to have in your etcd cluster
  # NOTE: -1 disables automatic etcd provisioning
  etcdSize: 1

  # Specify the PersistentVolumeClaim Spec which will be used as a storage for etcd
  # if it is not specified emptydir will be used
  etcdPVCSpec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi

  # This option allows you to annotate the ETCD Cluster that Vault Operator creates.
  # It's specifically to annotate the ETCD Cluster as 'clusterwide' for a cluster wide
  # ETCD Operator, however it can be used to set any arbitrary annotations on the ETCD Cluster.
  etcdAnnotations:
    etcd.database.coreos.com/scope: clusterwide

  # Annotations to be applied to the POD Specs
  etcdPodAnnotations:
    backup.velero.io/backup-volumes: "etcd-backup"

  # Describe where you would like to store the Vault unseal keys and root token.
  unsealConfig:
    kubernetes:
      secretNamespace: vault

  # A YAML representation of a final vault config file.
  # See https://www.vaultproject.io/docs/configuration/ for more information.
  config:
    storage:
      etcd:
        address: https://etcd-cluster:2379
        ha_enabled: "true"
    listener:
      tcp:
        address: "0.0.0.0:8200"
        tls_cert_file: /vault/tls/server.crt
        tls_key_file: /vault/tls/server.key
    api_addr: https://vault:8200
    telemetry:
      statsd_address: localhost:9125
    ui: true

  # See: https://github.com/banzaicloud/bank-vaults#example-external-vault-configuration for more details.
  externalConfig:
    policies:
      - name: allow_secrets
        rules: path "secret/*" {
          capabilities = ["create", "read", "update", "delete", "list"]
          }
    auth:
      - type: kubernetes
        roles:
          # Allow every pod in the default namespace to use the secret kv store
          - name: default
            bound_service_account_names: ["default", "vault-secrets-webhook"]
            bound_service_account_namespaces: ["default", "vault-wh", "mesh", "bookinfo"]
            policies: allow_secrets
            ttl: 1h
    secrets:
      - path: secret
        type: kv
        description: General secrets.
        options:
          version: 2

    # Allows writing some secrets to Vault (useful for development purposes).
    # See https://www.vaultproject.io/docs/secrets/kv/index.html for more information.
    startupSecrets:
      - type: kv
        path: secret/data/accounts/aws
        data:
          data:
            AWS_ACCESS_KEY_ID: secretId
            AWS_SECRET_ACCESS_KEY: s3cr3t
      - type: kv
        path: secret/sso/keycloak
        data:
          data:
            AUTH_SERVER_URL: https://url
            REALM: test

  vaultEnvsConfig:
    - name: VAULT_LOG_LEVEL
      value: debug

but looking into the vault-ui I'm not able to see the kubernetes auth and neither the startupSecrets or the kv secret created :S

@misanche I managed to reproduce the issue and it seems the LoadBalancer servicetype causes the problem. I put some small fixes in your cr-etcd-ha.yaml including bound_service_account_names, bound_service_account_namespaces and caNamespaces. The webhook will work properely if you deploy Vault with this CR:

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
spec:
  size: 2
  image: vault:1.1.0
  bankVaultsImage: banzaicloud/bank-vaults:latest

  # Specify the ServiceAccount where the Vault Pod and the Bank-Vaults configurer/unsealer is running
  serviceAccount: vault

  # Specify the Service's type where the Vault Service is exposed
  # serviceType: LoadBalancer

  # Specify how many nodes you would like to have in your etcd cluster
  # NOTE: -1 disables automatic etcd provisioning
  etcdSize: 1

  # Specify the PersistentVolumeClaim Spec which will be used as a storage for etcd
  # if it is not specified emptydir will be used
  etcdPVCSpec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi

  # This option allows you to annotate the ETCD Cluster that Vault Operator creates.
  # It's specifically to annotate the ETCD Cluster as 'clusterwide' for a cluster wide
  # ETCD Operator, however it can be used to set any arbitrary annotations on the ETCD Cluster.
  etcdAnnotations:
    etcd.database.coreos.com/scope: clusterwide

  # Annotations to be applied to the POD Specs
  etcdPodAnnotations:
    backup.velero.io/backup-volumes: "etcd-backup"

  caNamespaces:
    - "vault-wh"

  # Describe where you would like to store the Vault unseal keys and root token.
  unsealConfig:
    kubernetes:
      secretNamespace: vault

  # A YAML representation of a final vault config file.
  # See https://www.vaultproject.io/docs/configuration/ for more information.
  config:
    storage:
      etcd:
        address: https://etcd-cluster:2379
        ha_enabled: "true"
    listener:
      tcp:
        address: "0.0.0.0:8200"
        tls_cert_file: /vault/tls/server.crt
        tls_key_file: /vault/tls/server.key
    api_addr: https://vault:8200
    telemetry:
      statsd_address: localhost:9125
    ui: true

  # See: https://github.com/banzaicloud/bank-vaults#example-external-vault-configuration for more details.
  externalConfig:
    policies:
      - name: allow_secrets
        rules: path "secret/*" {
          capabilities = ["create", "read", "update", "delete", "list"]
          }
    auth:
      - type: kubernetes
        roles:
          # Allow every pod in the default namespace to use the secret kv store
          - name: default
            bound_service_account_names: ["default", "vault-wh-vault-secrets-webhook"]
            bound_service_account_namespaces: ["default", "vault-wh", "mesh", "bookinfo", "vault"]
            policies: allow_secrets
            ttl: 1h
    secrets:
      - path: secret
        type: kv
        description: General secrets.
        options:
          version: 2

    # Allows writing some secrets to Vault (useful for development purposes).
    # See https://www.vaultproject.io/docs/secrets/kv/index.html for more information.
    startupSecrets:
      - type: kv
        path: secret/data/accounts/aws
        data:
          data:
            AWS_ACCESS_KEY_ID: secretId
            AWS_SECRET_ACCESS_KEY: s3cr3t
      - type: kv
        path: secret/sso/keycloak
        data:
          data:
            AUTH_SERVER_URL: https://url
            REALM: test

  vaultEnvsConfig:
    - name: VAULT_LOG_LEVEL
      value: debug

We will dig into the loadBalancer issue.

Thanks @pbalogh-sa I will try tomorrow and let you know. Thank you for your effort

HI @pbalogh-sa I tried it and now it successfully creates the default roles, and auth. I was using istio with mtls, so I added a policy to disable mtls between my service and vault. that also worked. I'm able to retrieve the vault vars I think.

Now I have this problem:

Seems like the command to be executed it fails and the pod doesn't run the springboot app, I see this error:

2019/06/26 06:54:12 Received new Vault token
2019/06/26 06:54:12 Initial Vault token arrived
panic: interface conversion: interface {} is nil, not map[string]interface {}
 goroutine 1 [running]:
main.main()
    /build/cmd/vault-env/main.go:153 +0x29ca

Files: deployment.yaml:

{
  "kind": "Deployment",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "message-service-v1",
    "namespace": "mesh",
    "selfLink": "/apis/extensions/v1beta1/namespaces/mesh/deployments/message-service-v1",
    "uid": "c43f2b95-919b-11e9-a521-028284ad0f6c",
    "resourceVersion": "12886749",
    "generation": 40,
    "creationTimestamp": "2019-06-18T07:36:20Z",
    "labels": {
      "app": "message-service",
      "version": "v1"
    },
    "annotations": {
      "deployment.kubernetes.io/revision": "40",
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"message-service\",\"version\":\"v1\"},\"name\":\"message-service-v1\",\"namespace\":\"mesh\"},\"spec\":{\"replicas\":1,\"template\":{\"metadata\":{\"labels\":{\"app\":\"message-service\",\"version\":\"v1\"}},\"spec\":{\"containers\":[{\"image\":\"<imageUrl>\",\"imagePullPolicy\":\"Always\",\"name\":\"message-service\",\"ports\":[{\"containerPort\":8080}]}],\"imagePullSecrets\":[{\"name\":\"$KUBERNETES_SECRET_NAME\"}]}}}}\n"
    }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "app": "message-service",
        "version": "v1"
      }
    },
    "template": {
      "metadata": {
        "creationTimestamp": null,
        "labels": {
          "app": "message-service",
          "version": "v1"
        },
        "annotations": {
          "vault.security.banzaicloud.io/vault-addr": "https://vault.vault:8200",
          "vault.security.banzaicloud.io/vault-role": "default",
          "vault.security.banzaicloud.io/vault-skip-verify": "true"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "message-service",
            "image": "<imageUrl>",
            "command": [
              "java",
              "-Djava.security.egd=file:/dev/./urandom",
              "-Dspring.profiles.active=jdbc",
              "-Dapp.port=${app.port}",
              "-jar target/app.jar"
            ],
            "ports": [
              {
                "containerPort": 8080,
                "protocol": "TCP"
              }
            ],
            "env": [
              {
                "name": "JAEGER_SERVICE_NAME",
                "value": "message-service"
              },
              {
                "name": "JAEGER_ENDPOINT",
                "value": "http://jaeger-collector.istio-system.svc:14268/api/traces"
              },
              {
                "name": "JAEGER_PROPAGATION",
                "value": "b3"
              },
              {
                "name": "JAEGER_SAMPLER_TYPE",
                "value": "const"
              },
              {
                "name": "JAEGER_SAMPLER_PARAM",
                "value": "1"
              },
              {
                "name": "TEST",
                "value": "2"
              },
              {
                "name": "KC_AUTH_SERVER_URL",
                "value": "vault:secret/sso/keycloak#AUTH_SERVER_URL"
              },
              {
                "name": "KC_REALM",
                "value": "vault:secret/sso/keycloak#REALM"
              }
            ],
            "resources": {},
            "terminationMessagePath": "/dev/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "Always"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {},
        "schedulerName": "default-scheduler"
      }
    },
    "strategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "maxUnavailable": 1,
        "maxSurge": 1
      }
    },
    "revisionHistoryLimit": 10,
    "progressDeadlineSeconds": 2147483647
  },
  "status": {
    "observedGeneration": 40,
    "replicas": 1,
    "updatedReplicas": 1,
    "unavailableReplicas": 1,
    "conditions": [
      {
        "type": "Available",
        "status": "True",
        "lastUpdateTime": "2019-06-18T07:36:21Z",
        "lastTransitionTime": "2019-06-18T07:36:21Z",
        "reason": "MinimumReplicasAvailable",
        "message": "Deployment has minimum availability."
      }
    ],
    "collisionCount": 1
  }
}

The command that I added it's taken from the Dockerfile entry point:

FROM openjdk:8-jre-slim
VOLUME /tmp
ADD target/messagecenter-service-microservice-1.0.1-SNAPSHOT.jar target/app.jar
RUN sh -c 'touch target/app.jar'
EXPOSE 8080
ENV JAVA_OPTS=""
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-Dspring.profiles.active=jdbc", "-Dapp.port=${app.port}", "-jar","target/app.jar"]

I can see that the pod is created as:

{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "message-service-v1-795946588-hzlg2",
    "generateName": "message-service-v1-795946588-",
    "namespace": "mesh",
    "selfLink": "/api/v1/namespaces/mesh/pods/message-service-v1-795946588-hzlg2",
    "uid": "c453f2a9-97de-11e9-a521-028284ad0f6c",
    "resourceVersion": "12886801",
    "creationTimestamp": "2019-06-26T06:51:04Z",
    "labels": {
      "app": "message-service",
      "pod-template-hash": "351502144",
      "version": "v1"
    },
    "annotations": {
      "sidecar.istio.io/status": "{\"version\":\"d65154d7827c0ccdc7f441d8feec172b806755ad32f145ec2b91ffa177a6cefc\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}",
      "vault.security.banzaicloud.io/vault-addr": "https://vault.vault:8200",
      "vault.security.banzaicloud.io/vault-role": "default",
      "vault.security.banzaicloud.io/vault-skip-verify": "true"
    },
    "ownerReferences": [
      {
        "apiVersion": "apps/v1",
        "kind": "ReplicaSet",
        "name": "message-service-v1-795946588",
        "uid": "c44bf1dd-97de-11e9-a521-028284ad0f6c",
        "controller": true,
        "blockOwnerDeletion": true
      }
    ]
  },
  "spec": {
    "volumes": [
      {
        "name": "default-token-lcz6k",
        "secret": {
          "secretName": "default-token-lcz6k",
          "defaultMode": 420
        }
      },
      {
        "name": "istio-envoy",
        "emptyDir": {
          "medium": "Memory"
        }
      },
      {
        "name": "istio-certs",
        "secret": {
          "secretName": "istio.default",
          "defaultMode": 420,
          "optional": true
        }
      },
      {
        "name": "vault-env",
        "emptyDir": {
          "medium": "Memory"
        }
      },
      {
        "name": "vault-agent-config",
        "configMap": {
          "name": "message-service-v1-vault-agent-config",
          "defaultMode": 420
        }
      }
    ],
    "initContainers": [
      {
        "name": "vault-agent",
        "image": "vault:latest",
        "command": [
          "vault",
          "agent",
          "-config=/vault/agent/config.hcl"
        ],
        "env": [
          {
            "name": "VAULT_ADDR",
            "value": "https://vault.vault:8200"
          },
          {
            "name": "VAULT_SKIP_VERIFY",
            "value": "true"
          }
        ],
        "resources": {},
        "volumeMounts": [
          {
            "name": "vault-env",
            "mountPath": "/vault/"
          },
          {
            "name": "default-token-lcz6k",
            "readOnly": true,
            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
          },
          {
            "name": "vault-agent-config",
            "mountPath": "/vault/agent/"
          }
        ],
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "IfNotPresent",
        "securityContext": {
          "runAsUser": 100,
          "allowPrivilegeEscalation": false
        }
      },
      {
        "name": "copy-vault-env",
        "image": "banzaicloud/vault-env:latest",
        "command": [
          "sh",
          "-c",
          "cp /usr/local/bin/vault-env /vault/"
        ],
        "resources": {},
        "volumeMounts": [
          {
            "name": "vault-env",
            "mountPath": "/vault/"
          }
        ],
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "IfNotPresent",
        "securityContext": {
          "allowPrivilegeEscalation": false
        }
      },
      {
        "name": "istio-init",
        "image": "docker.io/istio/proxy_init:1.1.8",
        "args": [
          "-p",
          "15001",
          "-u",
          "1337",
          "-m",
          "REDIRECT",
          "-i",
          "*",
          "-x",
          "",
          "-b",
          "8080",
          "-d",
          "15020"
        ],
        "resources": {
          "limits": {
            "cpu": "100m",
            "memory": "50Mi"
          },
          "requests": {
            "cpu": "10m",
            "memory": "10Mi"
          }
        },
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "IfNotPresent",
        "securityContext": {
          "capabilities": {
            "add": [
              "NET_ADMIN"
            ]
          },
          "runAsUser": 0,
          "runAsNonRoot": false
        }
      }
    ],
    "containers": [
      {
        "name": "message-service",
        "image": "<imageUrl>",
        "command": [
          "/vault/vault-env"
        ],
        "args": [
          "java",
          "-Djava.security.egd=file:/dev/./urandom",
          "-Dspring.profiles.active=jdbc",
          "-Dapp.port=${app.port}",
          "-jar target/app.jar"
        ],
        "ports": [
          {
            "containerPort": 8080,
            "protocol": "TCP"
          }
        ],
        "env": [
          {
            "name": "JAEGER_SERVICE_NAME",
            "value": "message-service"
          },
          {
            "name": "JAEGER_ENDPOINT",
            "value": "http://jaeger-collector.istio-system.svc:14268/api/traces"
          },
          {
            "name": "JAEGER_PROPAGATION",
            "value": "b3"
          },
          {
            "name": "JAEGER_SAMPLER_TYPE",
            "value": "const"
          },
          {
            "name": "JAEGER_SAMPLER_PARAM",
            "value": "1"
          },
          {
            "name": "TEST",
            "value": "2"
          },
          {
            "name": "KC_AUTH_SERVER_URL",
            "value": "vault:secret/sso/keycloak#AUTH_SERVER_URL"
          },
          {
            "name": "KC_REALM",
            "value": "vault:secret/sso/keycloak#REALM"
          },
          {
            "name": "VAULT_ADDR",
            "value": "https://vault.vault:8200"
          },
          {
            "name": "VAULT_SKIP_VERIFY",
            "value": "true"
          },
          {
            "name": "VAULT_PATH",
            "value": "kubernetes"
          },
          {
            "name": "VAULT_ROLE",
            "value": "default"
          },
          {
            "name": "VAULT_IGNORE_MISSING_SECRETS",
            "value": "false"
          },
          {
            "name": "VAULT_ENV_PASSTHROUGH"
          },
          {
            "name": "VAULT_TOKEN_FILE",
            "value": "/vault/.vault-token"
          }
        ],
        "resources": {},
        "volumeMounts": [
          {
            "name": "default-token-lcz6k",
            "readOnly": true,
            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
          },
          {
            "name": "vault-env",
            "mountPath": "/vault/"
          }
        ],
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "Always"
      },
      {
        "name": "istio-proxy",
        "image": "docker.io/istio/proxyv2:1.1.8",
        "args": [
          "proxy",
          "sidecar",
          "--domain",
          "$(POD_NAMESPACE).svc.cluster.local",
          "--configPath",
          "/etc/istio/proxy",
          "--binaryPath",
          "/usr/local/bin/envoy",
          "--serviceCluster",
          "message-service.$(POD_NAMESPACE)",
          "--drainDuration",
          "45s",
          "--parentShutdownDuration",
          "1m0s",
          "--discoveryAddress",
          "istio-pilot.istio-system:15011",
          "--zipkinAddress",
          "zipkin.istio-system:9411",
          "--connectTimeout",
          "10s",
          "--proxyAdminPort",
          "15000",
          "--concurrency",
          "2",
          "--controlPlaneAuthPolicy",
          "MUTUAL_TLS",
          "--statusPort",
          "15020",
          "--applicationPorts",
          "8080"
        ],
        "ports": [
          {
            "name": "http-envoy-prom",
            "containerPort": 15090,
            "protocol": "TCP"
          }
        ],
        "env": [
          {
            "name": "POD_NAME",
            "valueFrom": {
              "fieldRef": {
                "apiVersion": "v1",
                "fieldPath": "metadata.name"
              }
            }
          },
          {
            "name": "POD_NAMESPACE",
            "valueFrom": {
              "fieldRef": {
                "apiVersion": "v1",
                "fieldPath": "metadata.namespace"
              }
            }
          },
          {
            "name": "INSTANCE_IP",
            "valueFrom": {
              "fieldRef": {
                "apiVersion": "v1",
                "fieldPath": "status.podIP"
              }
            }
          },
          {
            "name": "ISTIO_META_POD_NAME",
            "valueFrom": {
              "fieldRef": {
                "apiVersion": "v1",
                "fieldPath": "metadata.name"
              }
            }
          },
          {
            "name": "ISTIO_META_CONFIG_NAMESPACE",
            "valueFrom": {
              "fieldRef": {
                "apiVersion": "v1",
                "fieldPath": "metadata.namespace"
              }
            }
          },
          {
            "name": "ISTIO_META_INTERCEPTION_MODE",
            "value": "REDIRECT"
          },
          {
            "name": "ISTIO_METAJSON_ANNOTATIONS",
            "value": "{\"vault.security.banzaicloud.io/vault-addr\":\"https://vault.vault:8200\",\"vault.security.banzaicloud.io/vault-role\":\"default\",\"vault.security.banzaicloud.io/vault-skip-verify\":\"true\"}\n"
          },
          {
            "name": "ISTIO_METAJSON_LABELS",
            "value": "{\"app\":\"message-service\",\"pod-template-hash\":\"351502144\",\"version\":\"v1\"}\n"
          }
        ],
        "resources": {
          "limits": {
            "cpu": "2",
            "memory": "1Gi"
          },
          "requests": {
            "cpu": "100m",
            "memory": "128Mi"
          }
        },
        "volumeMounts": [
          {
            "name": "istio-envoy",
            "mountPath": "/etc/istio/proxy"
          },
          {
            "name": "istio-certs",
            "readOnly": true,
            "mountPath": "/etc/certs/"
          },
          {
            "name": "default-token-lcz6k",
            "readOnly": true,
            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
          }
        ],
        "readinessProbe": {
          "httpGet": {
            "path": "/healthz/ready",
            "port": 15020,
            "scheme": "HTTP"
          },
          "initialDelaySeconds": 1,
          "timeoutSeconds": 1,
          "periodSeconds": 2,
          "successThreshold": 1,
          "failureThreshold": 30
        },
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "imagePullPolicy": "IfNotPresent",
        "securityContext": {
          "runAsUser": 1337,
          "readOnlyRootFilesystem": true
        }
      }
    ],
    "restartPolicy": "Always",
    "terminationGracePeriodSeconds": 30,
    "dnsPolicy": "ClusterFirst",
    "serviceAccountName": "default",
    "serviceAccount": "default",
    "nodeName": "ip-10-50-36-122.us-east-2.compute.internal",
    "securityContext": {},
    "schedulerName": "default-scheduler",
    "tolerations": [
      {
        "key": "node.kubernetes.io/not-ready",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      },
      {
        "key": "node.kubernetes.io/unreachable",
        "operator": "Exists",
        "effect": "NoExecute",
        "tolerationSeconds": 300
      }
    ],
    "priority": 0
  },
  "status": {
    "phase": "Running",
    "conditions": [
      {
        "type": "Initialized",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-06-26T06:51:18Z"
      },
      {
        "type": "Ready",
        "status": "False",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-06-26T06:56:55Z",
        "reason": "ContainersNotReady",
        "message": "containers with unready status: [message-service]"
      },
      {
        "type": "ContainersReady",
        "status": "False",
        "lastProbeTime": null,
        "lastTransitionTime": null,
        "reason": "ContainersNotReady",
        "message": "containers with unready status: [message-service]"
      },
      {
        "type": "PodScheduled",
        "status": "True",
        "lastProbeTime": null,
        "lastTransitionTime": "2019-06-26T06:51:04Z"
      }
    ],
    "hostIP": "10.50.36.122",
    "podIP": "100.96.1.136",
    "startTime": "2019-06-26T06:51:04Z",
    "initContainerStatuses": [
      {
        "name": "vault-agent",
        "state": {
          "terminated": {
            "exitCode": 0,
            "reason": "Completed",
            "startedAt": "2019-06-26T06:51:15Z",
            "finishedAt": "2019-06-26T06:51:15Z",
            "containerID": "docker://179a49aa15d12364894781db91265426fe971d587c1b4a2a40500cd691d5a935"
          }
        },
        "lastState": {},
        "ready": true,
        "restartCount": 0,
        "image": "vault:latest",
        "imageID": "docker-pullable://vault@sha256:ae4853d4a14231889716ffb49d49b40d7e64f459810594ad7c679757ca66e0b9",
        "containerID": "docker://179a49aa15d12364894781db91265426fe971d587c1b4a2a40500cd691d5a935"
      },
      {
        "name": "copy-vault-env",
        "state": {
          "terminated": {
            "exitCode": 0,
            "reason": "Completed",
            "startedAt": "2019-06-26T06:51:16Z",
            "finishedAt": "2019-06-26T06:51:16Z",
            "containerID": "docker://1226921810e5c41326db61097fddee657a48e92cce8d3c6f9a7fb4d02566296d"
          }
        },
        "lastState": {},
        "ready": true,
        "restartCount": 0,
        "image": "banzaicloud/vault-env:latest",
        "imageID": "docker-pullable://banzaicloud/vault-env@sha256:669fd2f599ebb16bd46a25d4e8a68f679057396b12c911c9594a309ce3754128",
        "containerID": "docker://1226921810e5c41326db61097fddee657a48e92cce8d3c6f9a7fb4d02566296d"
      },
      {
        "name": "istio-init",
        "state": {
          "terminated": {
            "exitCode": 0,
            "reason": "Completed",
            "startedAt": "2019-06-26T06:51:17Z",
            "finishedAt": "2019-06-26T06:51:18Z",
            "containerID": "docker://c59bc3fa958b0b3092725a7e3411426ab2eaf7c7667d3a43693e487f1313875f"
          }
        },
        "lastState": {},
        "ready": true,
        "restartCount": 0,
        "image": "istio/proxy_init:1.1.8",
        "imageID": "docker-pullable://istio/proxy_init@sha256:e7a7e80afe26df29f132d70ed97d1d79de59a3b2c41d9a6f545dd96d6611e05f",
        "containerID": "docker://c59bc3fa958b0b3092725a7e3411426ab2eaf7c7667d3a43693e487f1313875f"
      }
    ],
    "containerStatuses": [
      {
        "name": "istio-proxy",
        "state": {
          "running": {
            "startedAt": "2019-06-26T06:51:19Z"
          }
        },
        "lastState": {},
        "ready": true,
        "restartCount": 0,
        "image": "istio/proxyv2:1.1.8",
        "imageID": "docker-pullable://istio/proxyv2@sha256:18c166cdd96f65d1fb7fb36cf914107fabd3b233db972b4352da72188b2de3f5",
        "containerID": "docker://88d6d26c6c580357d4623079284e98429fbd737a50819d738783ce48da8f8b2c"
      },
      {
        "name": "message-service",
        "state": {
          "waiting": {
            "reason": "CrashLoopBackOff",
            "message": "Back-off 5m0s restarting failed container=message-service pod=message-service-v1-795946588-hzlg2_mesh(c453f2a9-97de-11e9-a521-028284ad0f6c)"
          }
        },
        "lastState": {
          "terminated": {
            "exitCode": 2,
            "reason": "Error",
            "startedAt": "2019-06-26T06:56:54Z",
            "finishedAt": "2019-06-26T06:56:54Z",
            "containerID": "docker://6e66dca5c54bef0e1e8282627bb8650e2fe8249a049beb2529006ab0add55a53"
          }
        },
        "ready": false,
        "restartCount": 6,
        "image": "<imageUrl>",
        "imageID": "docker-pullable://<imageUrl>",
        "containerID": "docker://6e66dca5c54bef0e1e8282627bb8650e2fe8249a049beb2529006ab0add55a53"
      }
    ],
    "qosClass": "Burstable"
  }
}

Is this right, why the command is /vault/vault-env?

"command": [
          "/vault/vault-env"
        ],
        "args": [
          "java",
          "-Djava.security.egd=file:/dev/./urandom",
          "-Dspring.profiles.active=jdbc",
          "-Dapp.port=${app.port}",
          "-jar target/app.jar"
        ],

I have tried to split in between command and args:

"command": [
              "java"
            ],
            "args": [
              "-Djava.security.egd=file:/dev/./urandom",
              "-Dspring.profiles.active=jdbc",
              "-Dapp.port=${app.port}",
              "-jar target/app.jar"
            ],

Hi, @misanche the above mentioned command is right, webhook injects /vault/vault-env before your original command. Your error relates to something else: It seems there is an error around your vault secret, because it cannot access the metadata to a given key. Please update your CR with the following:

 - type: kv
        path: secret/data/sso/keycloak
        data:
          data:
            AUTH_SERVER_URL: https://url
            REALM: test

Also update your deployment environments with:

 {
            "name": "KC_AUTH_SERVER_URL",
            "value": "vault:secret/data/sso/keycloak#AUTH_SERVER_URL"
          },
          {
            "name": "KC_REALM",
            "value": "vault:secret/data/sso/keycloak#REALM"
          },

HI @baluchicken, I have fixed it but I removed the second data:

 - type: kv
        path: secret/data/sso/keycloak
        data:
          AUTH_SERVER_URL: https://url
          REALM: test

Looking into vault previously, it was created as:

{
  data: {
    "AUTH_SERVER_URL": "url"
    "REALM": "test"
  }
}

Now:

{
    "AUTH_SERVER_URL": "url"
    "REALM": "test"
}

Now I get this error:

2019/06/26 10:16:15 Received new Vault token
2019/06/26 10:16:15 Initial Vault token arrived
2019/06/26 10:16:15 Renewed Vault Token
Unrecognized option: -jar target/app.jar
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Could it be because we are in another path?

Regards

@baluchicken don't worry I solved it!

"command": [
          "/vault/vault-env"
        ],
        "args": [
          "java",
          "-Djava.security.egd=file:/dev/./urandom",
          "-Dspring.profiles.active=jdbc",
          "-Dapp.port=${app.port}",
          "-jar target/app.jar"
        ],

changed to:

"command": [
          "/vault/vault-env"
        ],
        "args": [
          "java",
          "-Djava.security.egd=file:/dev/./urandom",
          "-Dspring.profiles.active=jdbc",
          "-Dapp.port=${app.port}",
          "-jar",
          "target/app.jar"
        ],

@misanche the "-jar target/app.jar" argument is wrong try to use "-jar", "target/app.jar" instead

            "command": [
              "java",
              "-Djava.security.egd=file:/dev/./urandom",
              "-Dspring.profiles.active=jdbc",
              "-Dapp.port=${app.port}",
              "-jar",
              "target/app.jar"
            ],

Thanks guys, now works fine! sorry for all the questions! good job! I close the issue!

Your welcome. If you have new issue feel free to open a new ticket.

banzaicloud / banzai-charts

[vault-secrets-webhook] Failed to create pod #870