Closed misanche closed 5 years ago
Hi @misanche thanks for the issue. From this error message I would say the required Kubernetes secret is not found on the cluster/namespace where the pod wants to run.
Hi @baluchicken, thank you for your rapid response, It was working previously but this is how I install it:
kubectl create namespace vault-wh
kubectl label ns vault-wh name=vault-wh
helm upgrade --namespace vault-wh --install vault-wh banzaicloud-stable/vault-secrets-webhook --tls
Logs from the webhook pod:
time="2019-06-25T11:13:44Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:45Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:45Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:47Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:49Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:13:54Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:14:05Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:14:25Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:14:55Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:15:06Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:16:28Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:17:38Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:19:12Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:23:06Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found" time="2019-06-25T11:24:40Z" level=error msg="admission webhook error: cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \"$KUBERNETES_SECRET_NAME\" not found"
@misanche can you please check if you are using the latest version of the chart?
Sure @sagikazarmark,
from the deployment I can see the following annotation: "chart": "vault-secrets-webhook-0.3.17"
@misanche can you please check your pod specs which will be mutated by this webhook? Is there any pod which contains more than one ImagePullSecret?
@baluchicken yes it has.
I have removed now, the wh annotations and envs.
{
"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "message-service-v1",
"namespace": "mesh",
"selfLink": "/apis/extensions/v1beta1/namespaces/mesh/deployments/message-service-v1",
"uid": "c43f2b95-919b-11e9-a521-028284ad0f6c",
"resourceVersion": "12755080",
"generation": 30,
"creationTimestamp": "2019-06-18T07:36:20Z",
"labels": {
"app": "message-service",
"version": "v1"
},
"annotations": {
"deployment.kubernetes.io/revision": "30",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"message-service\",\"version\":\"v1\"},\"name\":\"message-service-v1\",\"namespace\":\"mesh\"},\"spec\":{\"replicas\":1,\"template\":{\"metadata\":{\"labels\":{\"app\":\"message-service\",\"version\":\"v1\"}},\"spec\":{\"containers\":[{\"image\":\"271960289458.dkr.ecr.us-east-2.amazonaws.com/reach-istio:message_5\",\"imagePullPolicy\":\"Always\",\"name\":\"message-service\",\"ports\":[{\"containerPort\":8080}]}],\"imagePullSecrets\":[{\"name\":\"$KUBERNETES_SECRET_NAME\"}]}}}}\n"
}
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": "message-service",
"version": "v1"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "message-service",
"version": "v1"
}
},
"spec": {
"containers": [
{
"name": "message-service",
"image": "<imageUrl>",
"ports": [
{
"containerPort": 8080,
"protocol": "TCP"
}
],
"env": [
{
"name": "JAEGER_SERVICE_NAME",
"value": "message-service"
},
{
"name": "JAEGER_ENDPOINT",
"value": "http://jaeger-collector.istio-system.svc:14268/api/traces"
},
{
"name": "JAEGER_PROPAGATION",
"value": "b3"
},
{
"name": "JAEGER_SAMPLER_TYPE",
"value": "const"
},
{
"name": "JAEGER_SAMPLER_PARAM",
"value": "1"
},
{
"name": "TEST",
"value": "2"
}
],
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "Always"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"imagePullSecrets": [
{
"name": "$KUBERNETES_SECRET_NAME"
}
],
"schedulerName": "default-scheduler"
}
},
"strategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": 1,
"maxSurge": 1
}
},
"revisionHistoryLimit": 10,
"progressDeadlineSeconds": 2147483647
},
"status": {
"observedGeneration": 30,
"replicas": 1,
"updatedReplicas": 1,
"readyReplicas": 1,
"availableReplicas": 1,
"conditions": [
{
"type": "Available",
"status": "True",
"lastUpdateTime": "2019-06-18T07:36:21Z",
"lastTransitionTime": "2019-06-18T07:36:21Z",
"reason": "MinimumReplicasAvailable",
"message": "Deployment has minimum availability."
}
]
}
}
If I remove the imagePullSecrets, I see the following error:
Error creating: Internal error occurred: failed calling admission webhook "pods.vault-secrets-webhook.admission.banzaicloud.com": an error on the server ("{\"response\":{\"uid\":\"16a60567-9749-11e9-a521-028284ad0f6c\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"cannot create client for registry: Get /v2/: unsupported protocol scheme \\\"\\\"\"}}}") has prevented the request from succeeding
I managed to find your problem. We just added a new feature to the webhook which kick in when there is no container commands and container args specified. https://github.com/banzaicloud/bank-vaults/blob/master/docs/mutating-webhook/README.md#using-charts-without-explicit-containercommand-and-containerargs
@baluchicken sorry, I don't have it very clear, what could I do now?
Regards
@misanche Is the "$KUBERNETES_SECRET_NAME" Kubernetes secret available in the pods namespace?
@baluchicken sec, it's not mine, I'm trying to deploy it without imagePullSecret or request more information.
thank you for your support
@baluchicken I have tried to remove and still doesn't work, but If I remove the imagePullSecrets and all the annotations and env vars for vault webhook, I'm able to deploy it.
{ "kind": "Pod", "apiVersion": "v1", "metadata": { "name": "message-service-v1-649ddcff9-6vjdq", "generateName": "message-service-v1-649ddcff9-", "namespace": "default", "selfLink": "/api/v1/namespaces/default/pods/message-service-v1-649ddcff9-6vjdq", "uid": "1fbe95f6-974d-11e9-a521-028284ad0f6c", "resourceVersion": "12770126", "creationTimestamp": "2019-06-25T13:28:31Z", "labels": { "app": "message-service", "pod-template-hash": "205887995", "version": "v1" }, "annotations": { "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu request for container message-service" }, "ownerReferences": [ { "apiVersion": "apps/v1", "kind": "ReplicaSet", "name": "message-service-v1-649ddcff9", "uid": "1fbcd8b8-974d-11e9-a521-028284ad0f6c", "controller": true, "blockOwnerDeletion": true } ] }, "spec": { "volumes": [ { "name": "default-token-qv2hq", "secret": { "secretName": "default-token-qv2hq", "defaultMode": 420 } } ], "containers": [ { "name": "message-service", "image": "<image>", "ports": [ { "containerPort": 8080, "protocol": "TCP" } ], "env": [ { "name": "JAEGER_SERVICE_NAME", "value": "message-service" }, { "name": "JAEGER_ENDPOINT", "value": "http://jaeger-collector.istio-system.svc:14268/api/traces" }, { "name": "JAEGER_PROPAGATION", "value": "b3" }, { "name": "JAEGER_SAMPLER_TYPE", "value": "const" }, { "name": "JAEGER_SAMPLER_PARAM", "value": "1" } ], "resources": { "requests": { "cpu": "100m" } }, "volumeMounts": [ { "name": "default-token-qv2hq", "readOnly": true, "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount" } ], "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "imagePullPolicy": "Always" } ], "restartPolicy": "Always", "terminationGracePeriodSeconds": 30, "dnsPolicy": "ClusterFirst", "serviceAccountName": "default", "serviceAccount": "default", "nodeName": "ip-10-50-85-254.us-east-2.compute.internal", "securityContext": {}, "schedulerName": "default-scheduler", "tolerations": [ { "key": "node.kubernetes.io/not-ready", "operator": "Exists", "effect": "NoExecute", "tolerationSeconds": 300 }, { "key": "node.kubernetes.io/unreachable", "operator": "Exists", "effect": "NoExecute", "tolerationSeconds": 300 } ], "priority": 0 }, "status": { "phase": "Running", "conditions": [ { "type": "Initialized", "status": "True", "lastProbeTime": null, "lastTransitionTime": "2019-06-25T13:28:31Z" }, { "type": "Ready", "status": "True", "lastProbeTime": null, "lastTransitionTime": "2019-06-25T13:28:49Z" }, { "type": "ContainersReady", "status": "True", "lastProbeTime": null, "lastTransitionTime": null }, { "type": "PodScheduled", "status": "True", "lastProbeTime": null, "lastTransitionTime": "2019-06-25T13:28:31Z" } ], "hostIP": "10.50.85.254", "podIP": "100.96.2.140", "startTime": "2019-06-25T13:28:31Z", "containerStatuses": [ { "name": "message-service", "state": { "running": { "startedAt": "2019-06-25T13:28:49Z" } }, "lastState": {}, "ready": true, "restartCount": 0, "image": "<image>", "imageID": "<imageId>", "containerID": "docker://3e29851e1350c31086fcdb4afcb15b50cdcf7c78d5927c9ec682f941ff127d34" } ], "qosClass": "Burstable" } }
I have removed imagePullSecrets because they are not needed, I get this error now:
2019/06/25 13:28:29 registry.ping url=/v2/
time="2019-06-25T13:28:29Z" level=error msg="admission webhook error: cannot create client for registry: Get /v2/: unsupported protocol scheme \"\""
I guess you are using a private repository for your images. If yes you have two options:
Hi @baluchicken I tried to follow your feedback and now I can see how the pod, but is stuck at Waiting: PodInitializing,
In the vault-agent init container I see this error:
2019-06-25T14:44:10.208Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT https://vault.vault:8200/v1/auth/kubernetes/login
Code: 400. Errors:
* missing client token" backoff=1.360566553
2019-06-25T14:44:11.569Z [INFO] auth.handler: authenticating
@baluchicken I don't know if this is the problem but I install the operator and the webhook in this way: `helm repo add banzaicloud-stable http://kubernetes-charts.banzaicloud.com/branch/master
helm repo update
helm install banzaicloud-stable/vault-operator --namespace=vault --tls --name=vault-operator
helm upgrade --install vault-operator banzaicloud-stable/vault-operator --set=etcd-operator.enabled=true --set=etcd-operator.etcdOperator.commandArgs.cluster-wide=true --namespace=vault --tls
kubectl apply -f rbac.yaml -n vault
kubectl apply -f cr-etcd-ha.yaml -n vault
kubectl create namespace vault-wh
kubectl label ns vault-wh name=vault-wh
helm upgrade --namespace vault-wh --install vault-wh banzaicloud-stable/vault-secrets-webhook --tls
rbac.yaml:
kind: ServiceAccount
apiVersion: v1
metadata:
name: vault
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vault-secrets
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- "*"
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vault-secrets
roleRef:
kind: Role
name: vault-secrets
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: vault
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault
namespace: vault
cr-etcd-ha.yaml:
apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
name: "vault"
spec:
size: 2
image: vault:1.1.0
bankVaultsImage: banzaicloud/bank-vaults:latest
# Specify the ServiceAccount where the Vault Pod and the Bank-Vaults configurer/unsealer is running
serviceAccount: vault
# Specify the Service's type where the Vault Service is exposed
serviceType: LoadBalancer
# Specify how many nodes you would like to have in your etcd cluster
# NOTE: -1 disables automatic etcd provisioning
etcdSize: 1
# Specify the PersistentVolumeClaim Spec which will be used as a storage for etcd
# if it is not specified emptydir will be used
etcdPVCSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
# This option allows you to annotate the ETCD Cluster that Vault Operator creates.
# It's specifically to annotate the ETCD Cluster as 'clusterwide' for a cluster wide
# ETCD Operator, however it can be used to set any arbitrary annotations on the ETCD Cluster.
etcdAnnotations:
etcd.database.coreos.com/scope: clusterwide
# Annotations to be applied to the POD Specs
etcdPodAnnotations:
backup.velero.io/backup-volumes: "etcd-backup"
# Describe where you would like to store the Vault unseal keys and root token.
unsealConfig:
kubernetes:
secretNamespace: vault
# A YAML representation of a final vault config file.
# See https://www.vaultproject.io/docs/configuration/ for more information.
config:
storage:
etcd:
address: https://etcd-cluster:2379
ha_enabled: "true"
listener:
tcp:
address: "0.0.0.0:8200"
tls_cert_file: /vault/tls/server.crt
tls_key_file: /vault/tls/server.key
api_addr: https://vault:8200
telemetry:
statsd_address: localhost:9125
ui: true
# See: https://github.com/banzaicloud/bank-vaults#example-external-vault-configuration for more details.
externalConfig:
policies:
- name: allow_secrets
rules: path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
auth:
- type: kubernetes
roles:
# Allow every pod in the default namespace to use the secret kv store
- name: default
bound_service_account_names: ["default", "vault-secrets-webhook"]
bound_service_account_namespaces: ["default", "vault-wh", "mesh", "bookinfo"]
policies: allow_secrets
ttl: 1h
secrets:
- path: secret
type: kv
description: General secrets.
options:
version: 2
# Allows writing some secrets to Vault (useful for development purposes).
# See https://www.vaultproject.io/docs/secrets/kv/index.html for more information.
startupSecrets:
- type: kv
path: secret/data/accounts/aws
data:
data:
AWS_ACCESS_KEY_ID: secretId
AWS_SECRET_ACCESS_KEY: s3cr3t
- type: kv
path: secret/sso/keycloak
data:
data:
AUTH_SERVER_URL: https://url
REALM: test
vaultEnvsConfig:
- name: VAULT_LOG_LEVEL
value: debug
but looking into the vault-ui I'm not able to see the kubernetes auth and neither the startupSecrets or the kv secret created :S
@misanche I managed to reproduce the issue and it seems the LoadBalancer servicetype causes the problem. I put some small fixes in your cr-etcd-ha.yaml including bound_service_account_names, bound_service_account_namespaces and caNamespaces. The webhook will work properely if you deploy Vault with this CR:
apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
name: "vault"
spec:
size: 2
image: vault:1.1.0
bankVaultsImage: banzaicloud/bank-vaults:latest
# Specify the ServiceAccount where the Vault Pod and the Bank-Vaults configurer/unsealer is running
serviceAccount: vault
# Specify the Service's type where the Vault Service is exposed
# serviceType: LoadBalancer
# Specify how many nodes you would like to have in your etcd cluster
# NOTE: -1 disables automatic etcd provisioning
etcdSize: 1
# Specify the PersistentVolumeClaim Spec which will be used as a storage for etcd
# if it is not specified emptydir will be used
etcdPVCSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
# This option allows you to annotate the ETCD Cluster that Vault Operator creates.
# It's specifically to annotate the ETCD Cluster as 'clusterwide' for a cluster wide
# ETCD Operator, however it can be used to set any arbitrary annotations on the ETCD Cluster.
etcdAnnotations:
etcd.database.coreos.com/scope: clusterwide
# Annotations to be applied to the POD Specs
etcdPodAnnotations:
backup.velero.io/backup-volumes: "etcd-backup"
caNamespaces:
- "vault-wh"
# Describe where you would like to store the Vault unseal keys and root token.
unsealConfig:
kubernetes:
secretNamespace: vault
# A YAML representation of a final vault config file.
# See https://www.vaultproject.io/docs/configuration/ for more information.
config:
storage:
etcd:
address: https://etcd-cluster:2379
ha_enabled: "true"
listener:
tcp:
address: "0.0.0.0:8200"
tls_cert_file: /vault/tls/server.crt
tls_key_file: /vault/tls/server.key
api_addr: https://vault:8200
telemetry:
statsd_address: localhost:9125
ui: true
# See: https://github.com/banzaicloud/bank-vaults#example-external-vault-configuration for more details.
externalConfig:
policies:
- name: allow_secrets
rules: path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
auth:
- type: kubernetes
roles:
# Allow every pod in the default namespace to use the secret kv store
- name: default
bound_service_account_names: ["default", "vault-wh-vault-secrets-webhook"]
bound_service_account_namespaces: ["default", "vault-wh", "mesh", "bookinfo", "vault"]
policies: allow_secrets
ttl: 1h
secrets:
- path: secret
type: kv
description: General secrets.
options:
version: 2
# Allows writing some secrets to Vault (useful for development purposes).
# See https://www.vaultproject.io/docs/secrets/kv/index.html for more information.
startupSecrets:
- type: kv
path: secret/data/accounts/aws
data:
data:
AWS_ACCESS_KEY_ID: secretId
AWS_SECRET_ACCESS_KEY: s3cr3t
- type: kv
path: secret/sso/keycloak
data:
data:
AUTH_SERVER_URL: https://url
REALM: test
vaultEnvsConfig:
- name: VAULT_LOG_LEVEL
value: debug
We will dig into the loadBalancer issue.
Thanks @pbalogh-sa I will try tomorrow and let you know. Thank you for your effort
HI @pbalogh-sa I tried it and now it successfully creates the default roles, and auth. I was using istio with mtls, so I added a policy to disable mtls between my service and vault. that also worked. I'm able to retrieve the vault vars I think.
Now I have this problem:
Seems like the command to be executed it fails and the pod doesn't run the springboot app, I see this error:
2019/06/26 06:54:12 Received new Vault token
2019/06/26 06:54:12 Initial Vault token arrived
panic: interface conversion: interface {} is nil, not map[string]interface {}
goroutine 1 [running]:
main.main()
/build/cmd/vault-env/main.go:153 +0x29ca
Files: deployment.yaml:
{
"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "message-service-v1",
"namespace": "mesh",
"selfLink": "/apis/extensions/v1beta1/namespaces/mesh/deployments/message-service-v1",
"uid": "c43f2b95-919b-11e9-a521-028284ad0f6c",
"resourceVersion": "12886749",
"generation": 40,
"creationTimestamp": "2019-06-18T07:36:20Z",
"labels": {
"app": "message-service",
"version": "v1"
},
"annotations": {
"deployment.kubernetes.io/revision": "40",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"message-service\",\"version\":\"v1\"},\"name\":\"message-service-v1\",\"namespace\":\"mesh\"},\"spec\":{\"replicas\":1,\"template\":{\"metadata\":{\"labels\":{\"app\":\"message-service\",\"version\":\"v1\"}},\"spec\":{\"containers\":[{\"image\":\"<imageUrl>\",\"imagePullPolicy\":\"Always\",\"name\":\"message-service\",\"ports\":[{\"containerPort\":8080}]}],\"imagePullSecrets\":[{\"name\":\"$KUBERNETES_SECRET_NAME\"}]}}}}\n"
}
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": "message-service",
"version": "v1"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "message-service",
"version": "v1"
},
"annotations": {
"vault.security.banzaicloud.io/vault-addr": "https://vault.vault:8200",
"vault.security.banzaicloud.io/vault-role": "default",
"vault.security.banzaicloud.io/vault-skip-verify": "true"
}
},
"spec": {
"containers": [
{
"name": "message-service",
"image": "<imageUrl>",
"command": [
"java",
"-Djava.security.egd=file:/dev/./urandom",
"-Dspring.profiles.active=jdbc",
"-Dapp.port=${app.port}",
"-jar target/app.jar"
],
"ports": [
{
"containerPort": 8080,
"protocol": "TCP"
}
],
"env": [
{
"name": "JAEGER_SERVICE_NAME",
"value": "message-service"
},
{
"name": "JAEGER_ENDPOINT",
"value": "http://jaeger-collector.istio-system.svc:14268/api/traces"
},
{
"name": "JAEGER_PROPAGATION",
"value": "b3"
},
{
"name": "JAEGER_SAMPLER_TYPE",
"value": "const"
},
{
"name": "JAEGER_SAMPLER_PARAM",
"value": "1"
},
{
"name": "TEST",
"value": "2"
},
{
"name": "KC_AUTH_SERVER_URL",
"value": "vault:secret/sso/keycloak#AUTH_SERVER_URL"
},
{
"name": "KC_REALM",
"value": "vault:secret/sso/keycloak#REALM"
}
],
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "Always"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"schedulerName": "default-scheduler"
}
},
"strategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": 1,
"maxSurge": 1
}
},
"revisionHistoryLimit": 10,
"progressDeadlineSeconds": 2147483647
},
"status": {
"observedGeneration": 40,
"replicas": 1,
"updatedReplicas": 1,
"unavailableReplicas": 1,
"conditions": [
{
"type": "Available",
"status": "True",
"lastUpdateTime": "2019-06-18T07:36:21Z",
"lastTransitionTime": "2019-06-18T07:36:21Z",
"reason": "MinimumReplicasAvailable",
"message": "Deployment has minimum availability."
}
],
"collisionCount": 1
}
}
The command that I added it's taken from the Dockerfile entry point:
FROM openjdk:8-jre-slim
VOLUME /tmp
ADD target/messagecenter-service-microservice-1.0.1-SNAPSHOT.jar target/app.jar
RUN sh -c 'touch target/app.jar'
EXPOSE 8080
ENV JAVA_OPTS=""
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-Dspring.profiles.active=jdbc", "-Dapp.port=${app.port}", "-jar","target/app.jar"]
I can see that the pod is created as:
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "message-service-v1-795946588-hzlg2",
"generateName": "message-service-v1-795946588-",
"namespace": "mesh",
"selfLink": "/api/v1/namespaces/mesh/pods/message-service-v1-795946588-hzlg2",
"uid": "c453f2a9-97de-11e9-a521-028284ad0f6c",
"resourceVersion": "12886801",
"creationTimestamp": "2019-06-26T06:51:04Z",
"labels": {
"app": "message-service",
"pod-template-hash": "351502144",
"version": "v1"
},
"annotations": {
"sidecar.istio.io/status": "{\"version\":\"d65154d7827c0ccdc7f441d8feec172b806755ad32f145ec2b91ffa177a6cefc\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}",
"vault.security.banzaicloud.io/vault-addr": "https://vault.vault:8200",
"vault.security.banzaicloud.io/vault-role": "default",
"vault.security.banzaicloud.io/vault-skip-verify": "true"
},
"ownerReferences": [
{
"apiVersion": "apps/v1",
"kind": "ReplicaSet",
"name": "message-service-v1-795946588",
"uid": "c44bf1dd-97de-11e9-a521-028284ad0f6c",
"controller": true,
"blockOwnerDeletion": true
}
]
},
"spec": {
"volumes": [
{
"name": "default-token-lcz6k",
"secret": {
"secretName": "default-token-lcz6k",
"defaultMode": 420
}
},
{
"name": "istio-envoy",
"emptyDir": {
"medium": "Memory"
}
},
{
"name": "istio-certs",
"secret": {
"secretName": "istio.default",
"defaultMode": 420,
"optional": true
}
},
{
"name": "vault-env",
"emptyDir": {
"medium": "Memory"
}
},
{
"name": "vault-agent-config",
"configMap": {
"name": "message-service-v1-vault-agent-config",
"defaultMode": 420
}
}
],
"initContainers": [
{
"name": "vault-agent",
"image": "vault:latest",
"command": [
"vault",
"agent",
"-config=/vault/agent/config.hcl"
],
"env": [
{
"name": "VAULT_ADDR",
"value": "https://vault.vault:8200"
},
{
"name": "VAULT_SKIP_VERIFY",
"value": "true"
}
],
"resources": {},
"volumeMounts": [
{
"name": "vault-env",
"mountPath": "/vault/"
},
{
"name": "default-token-lcz6k",
"readOnly": true,
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
},
{
"name": "vault-agent-config",
"mountPath": "/vault/agent/"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"runAsUser": 100,
"allowPrivilegeEscalation": false
}
},
{
"name": "copy-vault-env",
"image": "banzaicloud/vault-env:latest",
"command": [
"sh",
"-c",
"cp /usr/local/bin/vault-env /vault/"
],
"resources": {},
"volumeMounts": [
{
"name": "vault-env",
"mountPath": "/vault/"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"allowPrivilegeEscalation": false
}
},
{
"name": "istio-init",
"image": "docker.io/istio/proxy_init:1.1.8",
"args": [
"-p",
"15001",
"-u",
"1337",
"-m",
"REDIRECT",
"-i",
"*",
"-x",
"",
"-b",
"8080",
"-d",
"15020"
],
"resources": {
"limits": {
"cpu": "100m",
"memory": "50Mi"
},
"requests": {
"cpu": "10m",
"memory": "10Mi"
}
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"capabilities": {
"add": [
"NET_ADMIN"
]
},
"runAsUser": 0,
"runAsNonRoot": false
}
}
],
"containers": [
{
"name": "message-service",
"image": "<imageUrl>",
"command": [
"/vault/vault-env"
],
"args": [
"java",
"-Djava.security.egd=file:/dev/./urandom",
"-Dspring.profiles.active=jdbc",
"-Dapp.port=${app.port}",
"-jar target/app.jar"
],
"ports": [
{
"containerPort": 8080,
"protocol": "TCP"
}
],
"env": [
{
"name": "JAEGER_SERVICE_NAME",
"value": "message-service"
},
{
"name": "JAEGER_ENDPOINT",
"value": "http://jaeger-collector.istio-system.svc:14268/api/traces"
},
{
"name": "JAEGER_PROPAGATION",
"value": "b3"
},
{
"name": "JAEGER_SAMPLER_TYPE",
"value": "const"
},
{
"name": "JAEGER_SAMPLER_PARAM",
"value": "1"
},
{
"name": "TEST",
"value": "2"
},
{
"name": "KC_AUTH_SERVER_URL",
"value": "vault:secret/sso/keycloak#AUTH_SERVER_URL"
},
{
"name": "KC_REALM",
"value": "vault:secret/sso/keycloak#REALM"
},
{
"name": "VAULT_ADDR",
"value": "https://vault.vault:8200"
},
{
"name": "VAULT_SKIP_VERIFY",
"value": "true"
},
{
"name": "VAULT_PATH",
"value": "kubernetes"
},
{
"name": "VAULT_ROLE",
"value": "default"
},
{
"name": "VAULT_IGNORE_MISSING_SECRETS",
"value": "false"
},
{
"name": "VAULT_ENV_PASSTHROUGH"
},
{
"name": "VAULT_TOKEN_FILE",
"value": "/vault/.vault-token"
}
],
"resources": {},
"volumeMounts": [
{
"name": "default-token-lcz6k",
"readOnly": true,
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
},
{
"name": "vault-env",
"mountPath": "/vault/"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "Always"
},
{
"name": "istio-proxy",
"image": "docker.io/istio/proxyv2:1.1.8",
"args": [
"proxy",
"sidecar",
"--domain",
"$(POD_NAMESPACE).svc.cluster.local",
"--configPath",
"/etc/istio/proxy",
"--binaryPath",
"/usr/local/bin/envoy",
"--serviceCluster",
"message-service.$(POD_NAMESPACE)",
"--drainDuration",
"45s",
"--parentShutdownDuration",
"1m0s",
"--discoveryAddress",
"istio-pilot.istio-system:15011",
"--zipkinAddress",
"zipkin.istio-system:9411",
"--connectTimeout",
"10s",
"--proxyAdminPort",
"15000",
"--concurrency",
"2",
"--controlPlaneAuthPolicy",
"MUTUAL_TLS",
"--statusPort",
"15020",
"--applicationPorts",
"8080"
],
"ports": [
{
"name": "http-envoy-prom",
"containerPort": 15090,
"protocol": "TCP"
}
],
"env": [
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "POD_NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "INSTANCE_IP",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "status.podIP"
}
}
},
{
"name": "ISTIO_META_POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "ISTIO_META_CONFIG_NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "ISTIO_META_INTERCEPTION_MODE",
"value": "REDIRECT"
},
{
"name": "ISTIO_METAJSON_ANNOTATIONS",
"value": "{\"vault.security.banzaicloud.io/vault-addr\":\"https://vault.vault:8200\",\"vault.security.banzaicloud.io/vault-role\":\"default\",\"vault.security.banzaicloud.io/vault-skip-verify\":\"true\"}\n"
},
{
"name": "ISTIO_METAJSON_LABELS",
"value": "{\"app\":\"message-service\",\"pod-template-hash\":\"351502144\",\"version\":\"v1\"}\n"
}
],
"resources": {
"limits": {
"cpu": "2",
"memory": "1Gi"
},
"requests": {
"cpu": "100m",
"memory": "128Mi"
}
},
"volumeMounts": [
{
"name": "istio-envoy",
"mountPath": "/etc/istio/proxy"
},
{
"name": "istio-certs",
"readOnly": true,
"mountPath": "/etc/certs/"
},
{
"name": "default-token-lcz6k",
"readOnly": true,
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
}
],
"readinessProbe": {
"httpGet": {
"path": "/healthz/ready",
"port": 15020,
"scheme": "HTTP"
},
"initialDelaySeconds": 1,
"timeoutSeconds": 1,
"periodSeconds": 2,
"successThreshold": 1,
"failureThreshold": 30
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"runAsUser": 1337,
"readOnlyRootFilesystem": true
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"serviceAccountName": "default",
"serviceAccount": "default",
"nodeName": "ip-10-50-36-122.us-east-2.compute.internal",
"securityContext": {},
"schedulerName": "default-scheduler",
"tolerations": [
{
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
},
{
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
}
],
"priority": 0
},
"status": {
"phase": "Running",
"conditions": [
{
"type": "Initialized",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2019-06-26T06:51:18Z"
},
{
"type": "Ready",
"status": "False",
"lastProbeTime": null,
"lastTransitionTime": "2019-06-26T06:56:55Z",
"reason": "ContainersNotReady",
"message": "containers with unready status: [message-service]"
},
{
"type": "ContainersReady",
"status": "False",
"lastProbeTime": null,
"lastTransitionTime": null,
"reason": "ContainersNotReady",
"message": "containers with unready status: [message-service]"
},
{
"type": "PodScheduled",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2019-06-26T06:51:04Z"
}
],
"hostIP": "10.50.36.122",
"podIP": "100.96.1.136",
"startTime": "2019-06-26T06:51:04Z",
"initContainerStatuses": [
{
"name": "vault-agent",
"state": {
"terminated": {
"exitCode": 0,
"reason": "Completed",
"startedAt": "2019-06-26T06:51:15Z",
"finishedAt": "2019-06-26T06:51:15Z",
"containerID": "docker://179a49aa15d12364894781db91265426fe971d587c1b4a2a40500cd691d5a935"
}
},
"lastState": {},
"ready": true,
"restartCount": 0,
"image": "vault:latest",
"imageID": "docker-pullable://vault@sha256:ae4853d4a14231889716ffb49d49b40d7e64f459810594ad7c679757ca66e0b9",
"containerID": "docker://179a49aa15d12364894781db91265426fe971d587c1b4a2a40500cd691d5a935"
},
{
"name": "copy-vault-env",
"state": {
"terminated": {
"exitCode": 0,
"reason": "Completed",
"startedAt": "2019-06-26T06:51:16Z",
"finishedAt": "2019-06-26T06:51:16Z",
"containerID": "docker://1226921810e5c41326db61097fddee657a48e92cce8d3c6f9a7fb4d02566296d"
}
},
"lastState": {},
"ready": true,
"restartCount": 0,
"image": "banzaicloud/vault-env:latest",
"imageID": "docker-pullable://banzaicloud/vault-env@sha256:669fd2f599ebb16bd46a25d4e8a68f679057396b12c911c9594a309ce3754128",
"containerID": "docker://1226921810e5c41326db61097fddee657a48e92cce8d3c6f9a7fb4d02566296d"
},
{
"name": "istio-init",
"state": {
"terminated": {
"exitCode": 0,
"reason": "Completed",
"startedAt": "2019-06-26T06:51:17Z",
"finishedAt": "2019-06-26T06:51:18Z",
"containerID": "docker://c59bc3fa958b0b3092725a7e3411426ab2eaf7c7667d3a43693e487f1313875f"
}
},
"lastState": {},
"ready": true,
"restartCount": 0,
"image": "istio/proxy_init:1.1.8",
"imageID": "docker-pullable://istio/proxy_init@sha256:e7a7e80afe26df29f132d70ed97d1d79de59a3b2c41d9a6f545dd96d6611e05f",
"containerID": "docker://c59bc3fa958b0b3092725a7e3411426ab2eaf7c7667d3a43693e487f1313875f"
}
],
"containerStatuses": [
{
"name": "istio-proxy",
"state": {
"running": {
"startedAt": "2019-06-26T06:51:19Z"
}
},
"lastState": {},
"ready": true,
"restartCount": 0,
"image": "istio/proxyv2:1.1.8",
"imageID": "docker-pullable://istio/proxyv2@sha256:18c166cdd96f65d1fb7fb36cf914107fabd3b233db972b4352da72188b2de3f5",
"containerID": "docker://88d6d26c6c580357d4623079284e98429fbd737a50819d738783ce48da8f8b2c"
},
{
"name": "message-service",
"state": {
"waiting": {
"reason": "CrashLoopBackOff",
"message": "Back-off 5m0s restarting failed container=message-service pod=message-service-v1-795946588-hzlg2_mesh(c453f2a9-97de-11e9-a521-028284ad0f6c)"
}
},
"lastState": {
"terminated": {
"exitCode": 2,
"reason": "Error",
"startedAt": "2019-06-26T06:56:54Z",
"finishedAt": "2019-06-26T06:56:54Z",
"containerID": "docker://6e66dca5c54bef0e1e8282627bb8650e2fe8249a049beb2529006ab0add55a53"
}
},
"ready": false,
"restartCount": 6,
"image": "<imageUrl>",
"imageID": "docker-pullable://<imageUrl>",
"containerID": "docker://6e66dca5c54bef0e1e8282627bb8650e2fe8249a049beb2529006ab0add55a53"
}
],
"qosClass": "Burstable"
}
}
Is this right, why the command is /vault/vault-env?
"command": [
"/vault/vault-env"
],
"args": [
"java",
"-Djava.security.egd=file:/dev/./urandom",
"-Dspring.profiles.active=jdbc",
"-Dapp.port=${app.port}",
"-jar target/app.jar"
],
I have tried to split in between command and args:
"command": [
"java"
],
"args": [
"-Djava.security.egd=file:/dev/./urandom",
"-Dspring.profiles.active=jdbc",
"-Dapp.port=${app.port}",
"-jar target/app.jar"
],
Hi, @misanche the above mentioned command is right, webhook injects /vault/vault-env before your original command. Your error relates to something else: It seems there is an error around your vault secret, because it cannot access the metadata to a given key. Please update your CR with the following:
- type: kv
path: secret/data/sso/keycloak
data:
data:
AUTH_SERVER_URL: https://url
REALM: test
Also update your deployment environments with:
{
"name": "KC_AUTH_SERVER_URL",
"value": "vault:secret/data/sso/keycloak#AUTH_SERVER_URL"
},
{
"name": "KC_REALM",
"value": "vault:secret/data/sso/keycloak#REALM"
},
HI @baluchicken, I have fixed it but I removed the second data:
- type: kv
path: secret/data/sso/keycloak
data:
AUTH_SERVER_URL: https://url
REALM: test
Looking into vault previously, it was created as:
{
data: {
"AUTH_SERVER_URL": "url"
"REALM": "test"
}
}
Now:
{
"AUTH_SERVER_URL": "url"
"REALM": "test"
}
Now I get this error:
2019/06/26 10:16:15 Received new Vault token
2019/06/26 10:16:15 Initial Vault token arrived
2019/06/26 10:16:15 Renewed Vault Token
Unrecognized option: -jar target/app.jar
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Could it be because we are in another path?
Regards
@baluchicken don't worry I solved it!
"command": [
"/vault/vault-env"
],
"args": [
"java",
"-Djava.security.egd=file:/dev/./urandom",
"-Dspring.profiles.active=jdbc",
"-Dapp.port=${app.port}",
"-jar target/app.jar"
],
changed to:
"command": [
"/vault/vault-env"
],
"args": [
"java",
"-Djava.security.egd=file:/dev/./urandom",
"-Dspring.profiles.active=jdbc",
"-Dapp.port=${app.port}",
"-jar",
"target/app.jar"
],
@misanche the "-jar target/app.jar" argument is wrong try to use "-jar", "target/app.jar" instead
"command": [
"java",
"-Djava.security.egd=file:/dev/./urandom",
"-Dspring.profiles.active=jdbc",
"-Dapp.port=${app.port}",
"-jar",
"target/app.jar"
],
Thanks guys, now works fine! sorry for all the questions! good job! I close the issue!
Your welcome. If you have new issue feel free to open a new ticket.
Hi, I'm getting this issue, and I can see the following error from the replicaset:
Error creating: Internal error occurred: failed calling admission webhook "pods.vault-secrets-webhook.admission.banzaicloud.com": an error on the server ("{\"response\":{\"uid\":\"c55bece2-9738-11e9-a521-028284ad0f6c\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"cannot read imagePullSecrets '$KUBERNETES_SECRET_NAME': secrets \\"$KUBERNETES_SECRET_NAME\\" not found\"}}}") has prevented the request from succeeding