search

hashicorp / vault-k8s

First-class support for Vault and Kubernetes.
Mozilla Public License 2.0
790 stars 171 forks source link

Vault agent sidecar injector does not inject secrets (AWS). #110

Closed cassador closed 4 years ago

cassador commented 4 years ago

Hi.

I am running self managed k8s cluster in AWS. For networking i am using Calico. I followed this steps: https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/ But i am not able to see any credentials in app path /vault/secrets/helloworld. The only difference from the steps is that i was using it in default namespace instead of demo ( but i did the adjustments in policy etc. )

kubernetes version 1.17.3 helm version 3.1.2 I used following helm chart version: vault-0.4.0

I don't see any error only in kube-api server:

W0324 19:10:59.239317 1 dispatcher.go:168] Failed calling webhook, failing open vault.hashicorp.com: failed calling webhook "vault.hashicorp.com": Post https://vault-agent-injector-svc.default.svc:443/mutate?timeout=30s: context deadline exceeded E0324 19:10:59.239339 1 dispatcher.go:169] failed calling webhook "vault.hashicorp.com": Post https://vault-agent-injector-svc.default.svc:443/mutate?timeout=30s: context deadline exceeded I0324 19:10:59.247844 1 trace.go:116] Trace[430105269]: "Create" url:/api/v1/namespaces/default/pods,user-agent:kube-controller-manager/v1.17.3 (linux/amd64) kubernetes/06ad960/system:serviceaccount:kube-system:replicaset-controller,client:10.11.0.68 (started: 2020-03-24 19:10:29.232880841 +0000 UTC m=+1334648.401132298) (total time: 30.014915534s):

I allowed port 8080, 443, 6443 and other kubernetes recommended ports. I am running cluster of 3 nodes ( 1 master 2 worker nodes)

I did check and found similar issues opened like: https://github.com/hashicorp/vault-k8s/issues/32 But most of the issues are on GKE.

Thank you in advance.

woowil commented 4 years ago

I have similar problem:

https://github.com/hashicorp/vault-k8s/issues/114 https://github.com/hashicorp/vault-helm/issues/244

rchenzheng commented 4 years ago

I have similar problem:

114

hashicorp/vault-helm#244

Same here https://github.com/hashicorp/vault-helm/issues/335 but in GKE

rradecki-migo commented 4 years ago

Hi,

I have a similar issue as @cassador: aws eks kubernetes version: 1.17 eks.2 helm version: version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"} helm chart version: vault-0.7.0

All access from control plane to nodes and the other way around is permitted. Still upon pod creation (proper annotations are in place) no volume with the configured secret is added by vault injector.

Logs from the injector:

kubectl logs vault-agent-injector-6888874fb6-jmr6q -c sidecar-injector
2020-09-09T13:48:34.511Z [INFO]  handler: Starting handler..
Listening on ":8080"...
Updated certificate bundle received. Updating certs...

Logs for apiserver from CloudWatch with injector keyword:

I0909 14:37:01.834215       1 trace.go:116] Trace[1593070232]: "Call mutating webhook" configuration:vault-agent-injector-cfg,webhook:vault.hashicorp.com,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:36aed8bf-588d-4a4e-b347-31a2ce663bbb (started: 2020-09-09 14:36:31.83401446 +0000 UTC m=+1325315.894188766) (total time: 30.000159706s):
Trace[1593070232]: [30.000159706s] [30.000159706s] END

W0909 14:37:01.834279       1 dispatcher.go:168] Failed calling webhook, failing open vault.hashicorp.com: failed calling webhook "vault.hashicorp.com": Post https://vault-agent-injector-svc.vault-agent-injector-test.svc:443/mutate?timeout=30s: context deadline exceeded

E0909 14:37:01.834293       1 dispatcher.go:169] failed calling webhook "vault.hashicorp.com": Post https://vault-agent-injector-svc.vault-agent-injector-test.svc:443/mutate?timeout=30s: context deadline exceeded

I0909 14:37:01.845659       1 trace.go:116] Trace[1990763425]: "Create" url:/api/v1/namespaces/vault-agent-injector-test/pods,user-agent:kube-controller-manager/v1.17.9 (linux/amd64) kubernetes/4c69767/system:serviceaccount:kube-system:replicaset-controller,client:10.0.99.211 (started: 2020-09-09 14:36:31.825010539 +0000 UTC m=+1325315.885184829) (total time: 30.020618358s):
Trace[1990763425]: [30.009342818s] [30.009280364s] About to store object in database

I0909 14:37:05.072929       1 trace.go:116] Trace[1099609962]: "Call mutating webhook" configuration:vault-agent-injector-cfg,webhook:vault.hashicorp.com,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:f40ebc98-4b13-4ce4-aeb0-7aa1dba15e45 (started: 2020-09-09 14:36:35.072737329 +0000 UTC m=+1325319.132911683) (total time: 30.000146326s):
Trace[1099609962]: [30.000146326s] [30.000146326s] END

W0909 14:37:05.072983       1 dispatcher.go:168] Failed calling webhook, failing open vault.hashicorp.com: failed calling webhook "vault.hashicorp.com": Post https://vault-agent-injector-svc.vault-agent-injector-test.svc:443/mutate?timeout=30s: context deadline exceeded

E0909 14:37:05.073010       1 dispatcher.go:169] failed calling webhook "vault.hashicorp.com": Post https://vault-agent-injector-svc.vault-agent-injector-test.svc:443/mutate?timeout=30s: context deadline exceeded

I0909 14:37:05.077528       1 trace.go:116] Trace[1613249883]: "Create" url:/api/v1/namespaces/vault-agent-injector-test/pods,user-agent:kube-controller-manager/v1.17.9 (linux/amd64) kubernetes/4c69767/system:serviceaccount:kube-system:replicaset-controller,client:10.0.99.211 (started: 2020-09-09 14:36:35.070060903 +0000 UTC m=+1325319.130235197) (total time: 30.007437998s):
Trace[1613249883]: [30.003004293s] [30.002952426s] About to store object in database

Can someone help us with this one?

Thanks!

tvoran commented 4 years ago

Hi @cassador and @rradecki-migo, for questions like these it would help to have more details, things like:

(And also, a good place to get answers is on our discussion forum, as it gets more visibility from experienced users than the issue tracker.)

@woowil and @rchenzheng, it looks like your respective issues (https://github.com/hashicorp/vault-helm/issues/244, https://github.com/hashicorp/vault-helm/issues/335) have already been resolved.

rradecki-migo commented 4 years ago

Hi @tvoran,

Thanks for taking a look. Sure, please check details added below. vault injector logs

logs vault-agent-injector-6888874fb6-jmr6q
2020-09-09T13:48:34.511Z [INFO]  handler: Starting handler..
Listening on ":8080"...
Updated certificate bundle received. Updating certs...
Updated certificate bundle received. Updating certs...
Updated certificate bundle received. Updating certs...
Updated certificate bundle received. Updating certs...
Updated certificate bundle received. Updating certs...
Updated certificate bundle received. Updating certs...
Updated certificate bundle received. Updating certs...

vault injector deployment config - I updated AGENT_INJECT_LOG_LEVEL to debug but after pod recreation amount of log messages did not increase

get deploy vault-agent-injector -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  creationTimestamp: "2020-09-09T11:36:33Z"
  generation: 2
  labels:
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vault-agent-injector
    component: webhook
  name: vault-agent-injector
  namespace: vault-agent-injector-test
  resourceVersion: "52523260"
  selfLink: /apis/apps/v1/namespaces/vault-agent-injector-test/deployments/vault-agent-injector
  uid: 01fcde93-2d78-434f-a90c-70da1c7eed84
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: vault
      app.kubernetes.io/name: vault-agent-injector
      component: webhook
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: vault
        app.kubernetes.io/name: vault-agent-injector
        component: webhook
    spec:
      containers:
      - args:
        - agent-inject
        - 2>&1
        env:
        - name: AGENT_INJECT_LISTEN
          value: :8080
        - name: AGENT_INJECT_LOG_LEVEL
          value: debug
        - name: AGENT_INJECT_VAULT_ADDR
          value: https://OUR_VAULT_URL:8200
        - name: AGENT_INJECT_VAULT_AUTH_PATH
          value: auth/kubernetes
        - name: AGENT_INJECT_VAULT_IMAGE
          value: vault:1.5.2
        - name: AGENT_INJECT_TLS_AUTO
          value: vault-agent-injector-cfg
        - name: AGENT_INJECT_TLS_AUTO_HOSTS
          value: vault-agent-injector-svc,vault-agent-injector-svc.vault-agent-injector-test,vault-agent-injector-svc.vault-agent-injector-test.svc
        - name: AGENT_INJECT_LOG_FORMAT
          value: standard
        - name: AGENT_INJECT_REVOKE_ON_SHUTDOWN
          value: "false"
        image: hashicorp/vault-k8s:0.5.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /health/ready
            port: 8080
            scheme: HTTPS
          initialDelaySeconds: 1
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 5
        name: sidecar-injector
        readinessProbe:
          failureThreshold: 2
          httpGet:
            path: /health/ready
            port: 8080
            scheme: HTTPS
          initialDelaySeconds: 2
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 5
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 100
      serviceAccount: vault-agent-injector
      serviceAccountName: vault-agent-injector
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-09-09T11:36:38Z"
    lastUpdateTime: "2020-09-09T11:36:38Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2020-09-09T11:36:33Z"
    lastUpdateTime: "2020-09-09T13:48:47Z"
    message: ReplicaSet "vault-agent-injector-6888874fb6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

vault agent logs - not available due to the fact that no vault agent was injected by the mutating webhook my app deployment

kubectl get deploy devwebapp -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"devwebapp"},"name":"devwebapp","namespace":"vault-agent-injector-test"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"devwebapp"}},"template":{"metadata":{"labels":{"app":"devwebapp"}},"spec":{"containers":[{"env":[{"name":"VAULT_ADDR","value":"https://OUR_VAULT_URL:8200"}],"image":"burtlo/devwebapp-ruby:k8s","imagePullPolicy":"IfNotPresent","name":"app"}],"serviceAccountName":"internal-app"}}}}
  creationTimestamp: "2020-09-09T14:36:31Z"
  generation: 2
  labels:
    app: devwebapp
  name: devwebapp
  namespace: vault-agent-injector-test
  resourceVersion: "52532325"
  selfLink: /apis/apps/v1/namespaces/vault-agent-injector-test/deployments/devwebapp
  uid: f57d8928-a4c0-430f-89ad-5de654bb1562
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: devwebapp
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-credentials.txt: secret/data/devwebapp/config
        vault.hashicorp.com/role: devweb-app
      creationTimestamp: null
      labels:
        app: devwebapp
    spec:
      containers:
      - env:
        - name: VAULT_ADDR
          value: https://OUR_VAULT_URL:8200
        image: burtlo/devwebapp-ruby:k8s
        imagePullPolicy: IfNotPresent
        name: app
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: internal-app
      serviceAccountName: internal-app
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-09-09T14:37:03Z"
    lastUpdateTime: "2020-09-09T14:37:03Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2020-09-09T14:36:31Z"
    lastUpdateTime: "2020-09-09T14:37:07Z"
    message: ReplicaSet "devwebapp-85c4d8d898" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

helm values

helm ls
NAME    NAMESPACE                       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
vault   vault-agent-injector-test       1               2020-09-09 13:36:32.694273 +0200 CEST   deployed        vault-0.7.0     1.5.2

helm get values vault
USER-SUPPLIED VALUES:
injector:
  externalVaultAddr: https://OUR_VAULT_URL:8200

requested describe commands

kubectl describe deployment devwebapp
Name:                   devwebapp
Namespace:              vault-agent-injector-test
CreationTimestamp:      Wed, 09 Sep 2020 16:36:31 +0200
Labels:                 app=devwebapp
Annotations:            deployment.kubernetes.io/revision: 2
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"devwebapp"},"name":"devwebapp","namespace":"vaul...
Selector:               app=devwebapp
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=devwebapp
  Annotations:      vault.hashicorp.com/agent-inject: true
                    vault.hashicorp.com/agent-inject-secret-credentials.txt: secret/data/devwebapp/config
                    vault.hashicorp.com/role: devweb-app
  Service Account:  internal-app
  Containers:
   app:
    Image:      burtlo/devwebapp-ruby:k8s
    Port:       <none>
    Host Port:  <none>
    Environment:
      VAULT_ADDR:  https://OUR_VAULT_URL:8200
    Mounts:        <none>
  Volumes:         <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   devwebapp-85c4d8d898 (1/1 replicas created)
Events:          <none>
kubectl describe rs devwebapp
Name:           devwebapp-6646c98b78
Namespace:      vault-agent-injector-test
Selector:       app=devwebapp,pod-template-hash=6646c98b78
Labels:         app=devwebapp
                pod-template-hash=6646c98b78
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 1
Controlled By:  Deployment/devwebapp
Replicas:       0 current / 0 desired
Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=devwebapp
                    pod-template-hash=6646c98b78
  Service Account:  internal-app
  Containers:
   app:
    Image:      burtlo/devwebapp-ruby:k8s
    Port:       <none>
    Host Port:  <none>
    Environment:
      VAULT_ADDR:  https://OUR_VAULT_URL:8200
    Mounts:        <none>
  Volumes:         <none>
Events:            <none>

Name:           devwebapp-85c4d8d898
Namespace:      vault-agent-injector-test
Selector:       app=devwebapp,pod-template-hash=85c4d8d898
Labels:         app=devwebapp
                pod-template-hash=85c4d8d898
Annotations:    deployment.kubernetes.io/desired-replicas: 1
                deployment.kubernetes.io/max-replicas: 2
                deployment.kubernetes.io/revision: 2
Controlled By:  Deployment/devwebapp
Replicas:       1 current / 1 desired
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=devwebapp
                    pod-template-hash=85c4d8d898
  Annotations:      vault.hashicorp.com/agent-inject: true
                    vault.hashicorp.com/agent-inject-secret-credentials.txt: secret/data/devwebapp/config
                    vault.hashicorp.com/role: devweb-app
  Service Account:  internal-app
  Containers:
   app:
    Image:      burtlo/devwebapp-ruby:k8s
    Port:       <none>
    Host Port:  <none>
    Environment:
      VAULT_ADDR:  https://OUR_VAULT_URL:8200
    Mounts:        <none>
  Volumes:         <none>
Events:            <none>

Do you need anything else?

cassador commented 4 years ago

Hi @tvoran . I had similar setup as @rradecki-migo with following the tutorial from hashicorp documentation. Currently i am not able to provide the data as i was clearing the whole system , but i should be able to provide you the information within upcoming days.

cassador commented 4 years ago

@tvoran i was able to resolve the issue. In my case it was not working because of miss-configured TLS certificate.

Currently i am facing other issue if you might help and that it https://www.vaultproject.io/docs/platform/k8s/injector/examples#environment-variable-example To set as OS it's not working trying to follow:

` vault.hashicorp.com/agent-inject-secret-config: "secret/data/web"

Environment variable export template

    vault.hashicorp.com/agent-inject-template-config: |
      {{ with secret "secret/data/web" -}}
        export api_key="{{ .Data.data.payments_api_key }}"
      {{- end }}
spec:
  serviceAccountName: web
  containers:
    - name: web
      image: alpine:latest
      args: ["sh", "-c", "source /vault/secrets/config && <entrypoint script>"]`
tvoran commented 4 years ago

@rradecki-migo Hmm, I'm not able to reproduce your issue. I tried your deployment on EKS, and it was injecting the vault-agent containers as expected. Even with a non-existent external vault address it still injects the containers, just doesn't get past the init container:

$ kubectl get pod -n vault-agent-injector-test 
NAME                                    READY   STATUS     RESTARTS   AGE
devwebapp-857668fb7d-5bjnq              0/2     Init:0/1   0          46m
vault-agent-injector-67c596f8ff-swscl   1/1     Running    0          46m

I'm using helm 3.3.3 if that makes any difference. Maybe take a look at the mutating webhook too, should look something like this:

```console $ k get mutatingwebhookconfigurations.admissionregistration.k8s.io vault-agent-injector-cfg -o yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: annotations: meta.helm.sh/release-name: vault meta.helm.sh/release-namespace: vault-agent-injector-test creationTimestamp: "2020-09-23T05:56:56Z" generation: 2 labels: app.kubernetes.io/instance: vault app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: vault-agent-injector name: vault-agent-injector-cfg resourceVersion: "3619" selfLink: /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/vault-agent-injector-cfg uid: 5b5e63cc-d62f-4143-b810-1cf5d7fefe2c webhooks: - admissionReviewVersions: - v1beta1 clientConfig: caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNSekNDQWU2Z0F3SUJBZ0lVSS94SWprbjExSzZ6c1lqdTViTDdib1pSREFrd0NnWUlLb1pJemowRUF3SXcKR2pFWU1CWUdBMVVFQXhNUFFXZGxiblFnU1c1cVpXTjBJRU5CTUI0WERUSXdNRGt5TXpBMU5UWXdNRm9YRFRNdwpNRGt5TVRBMU5UY3dNRm93R2pFWU1CWUdBMVVFQXhNUFFXZGxiblFnU1c1cVpXTjBJRU5CTUZrd0V3WUhLb1pJCnpqMENBUVlJS29aSXpqMERBUWNEUWdBRUtqWUV1b3lQYmJpY2MybHhNVis2Rmp4ZEl5SjltRzBmQkhYaDZ1WVEKLzEvNzJ0U1FRVFN3NWNPM29pQnY1aHlBWE9YZU5UZnlaZUpPT1lubERpeVlPYU9DQVJBd2dnRU1NQTRHQTFVZApEd0VCL3dRRUF3SUNoREFUQmdOVkhTVUVEREFLQmdnckJnRUZCUWNEQVRBUEJnTlZIUk1CQWY4RUJUQURBUUgvCk1HZ0dBMVVkRGdSaEJGOHdaVHBrT0RveE56cGlaam8wTnpvME9Eb3paRG93WlRveE5UbzFOVG93TlRvM05UcGkKTkRwbU5Ub3dPVG8zTXpvNU5Ub3paRHBsTURwaE5UcGxOVG94WmpwallqcG1ZenBtTmpvMk1Eb3lNanBoWWpvMgpaanBsWkRvek5UbzJaREJxQmdOVkhTTUVZekJoZ0Y4d1pUcGtPRG94TnpwaVpqbzBOem8wT0RvelpEb3daVG94Ck5UbzFOVG93TlRvM05UcGlORHBtTlRvd09UbzNNem81TlRvelpEcGxNRHBoTlRwbE5Ub3haanBqWWpwbVl6cG0KTmpvMk1Eb3lNanBoWWpvMlpqcGxaRG96TlRvMlpEQUtCZ2dxaGtqT1BRUURBZ05IQURCRUFpQlE4Q3J4MUM2ZAorWERHWUxhNk9wZEw5KzZScm5Pa1NuWTZxcllqYUZoZWhnSWdZVkVlYTRJNGF3akhTeGJUUmhDZWJVMXdEUEVmCnh4aEhBaTlLUklTT1BVQT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= service: name: vault-agent-injector-svc namespace: vault-agent-injector-test path: /mutate port: 443 failurePolicy: Ignore matchPolicy: Exact name: vault.hashicorp.com namespaceSelector: {} objectSelector: {} reinvocationPolicy: Never rules: - apiGroups: - "" apiVersions: - v1 operations: - CREATE - UPDATE resources: - pods scope: '*' sideEffects: Unknown timeoutSeconds: 30 ```
tvoran commented 4 years ago

@cassador Great! Glad you got the TLS issue sorted out.

Your environment variable question would be a good one for the discuss forum: https://discuss.hashicorp.com/c/vault

rradecki-migo commented 4 years ago

@tvoran I checked my EKS setup on various levels and it seems that the problem is related to Calico CNI usage in the cluster. When it is used there is no option to start Calico pods on master nodes and as a result they are not able to interact with pods spawned on worker nodes through virtualized network. Info is available in https://docs.projectcalico.org/getting-started/kubernetes/managed-public-cloud/eks:

Note: Calico networking cannot currently be installed on the EKS control plane nodes. As a result the control plane nodes will not be able to initiate network connections to Calico pods. (This is a general limitation of EKS’s custom networking support, not specific to Calico.) As a workaround, trusted pods that require control plane nodes to connect to them, such as those implementing admission controller webhooks, can include hostNetwork:true in their pod spec. See the Kuberentes API pod spec definition for more information on this setting.

Is there maybe a plan to provide hostNetwork support for Vault agent injector?

tvoran commented 4 years ago

@rradecki-migo Nice find! Would you mind opening a feature request on vault-helm for adding hostNetwork support to the injector deployment?

https://github.com/hashicorp/vault-helm/issues/new/choose

tvoran commented 4 years ago

Closing since the original problem is resolved.

pksurferdad commented 4 years ago

@cassador i know this issue is closed, but can you share how you resolved your TLS issue https://github.com/hashicorp/vault-k8s/issues/110#issuecomment-696940650? I'm running Vault on a stand-alone EKS cluster and vault-agent-init doesn't complete and remains in a started state. I'm pretty sure it's some AWS networking issue.

cassador commented 4 years ago

@pksurferdad sure. I followed the following 3 series of videos from Marcel Dumpers: https://www.youtube.com/watch?v=jEUyKjEatWg

He is pointing the procedure for his github in the comments but for the TLS i followed this section: https://github.com/marcel-dempers/docker-development-youtube-series/blob/master/hashicorp/vault/tls/ssl_generate_self_signed.txt

I did every step he mentioned on his 3 videos and following this generation of TLS and was able to get it standalone running.

I hope this will help you :)

Sorry for late reply just was little overwhelmed with work so didn't get a chance to look into e-mails and notification's

pksurferdad commented 4 years ago

thx for your response @cassador. i'll give a shot and let you know how it goes.

pksurferdad commented 4 years ago

thx @cassador ... i got the tls cert working on those videos are awesome! it also led me to get this fully working now getting a secret from vault running on k8s.

cassador commented 4 years ago

@pksurferdad glad to hear that :). I was also unlucky at the start so i even wrote to Marcel Dumpers on facebook and he replied that if i did follow all three videos and he sent me an link for each one and then i realized that I skipped one :D .

byronmansfield commented 3 years ago

@cassador can I ask what was the misconfigured TLS cert issue you had. I think I'm probably having a similiar issue. I'm using the helm chart with TLS, auto-unseal, and HA turned on with raft as the backend on eks. I have searched everywhere trying to figure out why I'm seeing the error in the agent injector pod of http: TLS handshake error from 172.16.3.210:38022: remote error: tls: bad certificate. I have followed this tutorial to the tee and I can't even get the test pod to work. I also can't get the test container to authorize either. I'm fairly certain it's the caBundle part that I have wrong. In the example it shows just piping it to base64. But this comes out as multi-lines. And it fails to deploy this way for me. So I pipe that out to tr -d '\n' and I still get this tls bad cert.