Not able to have secrets on /tmp/secret

[victorarbuesmallada]

Hello.

We deployed the secret-inject into one of our EKS clusters and, even though on the secret inject logs we seem to be pulling secrets correctly, there's no secret saved into /tmp/secret at all. Both the secret exists and the service account is using a valid role (as you can see on the logs, we are getting a valid but empty response on pods.go:157).

I0717 09:53:38.080245       1 main.go:81] handling request: {"kind":"AdmissionReview","apiVersion":"admission.k8s.io/v1beta1","request":{"uid":"5f51ec5e-758a-47d0-9dd4-17156b0b54bb","kind":{"group":"","version":"v1","kind":"Pod"},"resource":{"group":"","version":"v1","resource":"pods"},"requestKind":{"group":"","version":"v1","kind":"Pod"},"requestResource":{"group":"","version":"v1","resource":"pods"},"namespace":"default","operation":"CREATE","userInfo":{"username":"system:serviceaccount:kube-system:replicaset-controller","uid":"b5552644-6aa3-11ea-b53c-0ab35cafaebc","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"object":{"kind":"Pod","apiVersion":"v1","metadata":{"generateName":"webserver-7b56c5866-","creationTimestamp":null,"labels":{"pod-template-hash":"7b56c5866","run":"webserver"},"annotations":{"kubernetes.io/psp":"eks.privileged","secrets.k8s.aws/secret-arn":"arn:aws:secretsmanager:eu-west-2:1234567:secret:foo-EYL7CW","secrets.k8s.aws/sidecarInjectorWebhook":"enabled","sidecar.istio.io/inject":"false"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"webserver-7b56c5866","uid":"ae04dfa2-a7f9-401a-bfb6-50b328229506","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"service-account-token-6ntkj","secret":{"secretName":"service-account-token-6ntkj"}}],"containers":[{"name":"webserver","image":"busybox:1.28","command":["sh","-c","echo $(cat /tmp/secret) \u0026\u0026 sleep 3600"],"resources":{},"volumeMounts":[{"name":"service-account-token-6ntkj","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"service-account","serviceAccount":"service-account","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{}},"oldObject":null,"dryRun":false,"options":{"kind":"CreateOptions","apiVersion":"meta.k8s.io/v1"}}}
I0717 09:53:38.080460       1 pods.go:157] &AdmissionResponse{UID:,Allowed:true,Result:nil,Patch:*[91 10 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 123 34 111 112 34 58 34 97 100 100 34 44 34 112 97 116 104 34 58 34 47 115 112 101 99 47 105 110 105 116 67 111 110 116 97 105 110 101 114 115 34 44 34 118 97 108 117 101 34 58 91 123 34 105 109 97 103 101 34 58 34 100 111 99 107 101 114 46 105 111 47 97 109 97 122 111 110 47 97 119 115 45 115 101 99 114 101 116 115 45 109 97 110 97 103 101 114 45 115 101 99 114 101 116 45 115 105 100 101 99 97 114 58 118 48 46 49 46 49 34 44 34 110 97 109 101 34 58 34 115 101 99 114 101 116 115 45 105 110 105 116 45 99 111 110 116 97 105 110 101 114 34 44 34 118 111 108 117 109 101 77 111 117 110 116 115 34 58 91 123 34 110 97 109 101 34 58 34 115 101 99 114 101 116 45 118 111 108 34 44 34 109 111 117 110 116 80 97 116 104 34 58 34 47 116 109 112 34 125 93 44 34 101 110 118 34 58 91 123 34 110 97 109 101 34 58 32 34 83 69 67 82 69 84 95 65 82 78 34 44 34 118 97 108 117 101 70 114 111 109 34 58 32 123 34 102 105 101 108 100 82 101 102 34 58 32 123 34 102 105 101 108 100 80 97 116 104 34 58 32 34 109 101 116 97 100 97 116 97 46 97 110 110 111 116 97 116 105 111 110 115 91 39 115 101 99 114 101 116 115 46 107 56 115 46 97 119 115 47 115 101 99 114 101 116 45 97 114 110 39 93 34 125 125 125 93 44 34 114 101 115 111 117 114 99 101 115 34 58 123 125 125 93 125 44 123 34 111 112 34 58 34 97 100 100 34 44 34 112 97 116 104 34 58 34 47 115 112 101 99 47 118 111 108 117 109 101 115 47 45 34 44 34 118 97 108 117 101 34 58 123 34 101 109 112 116 121 68 105 114 34 58 32 123 34 109 101 100 105 117 109 34 58 32 34 77 101 109 111 114 121 34 125 44 34 110 97 109 101 34 58 32 34 115 101 99 114 101 116 45 118 111 108 34 125 125 44 123 34 111 112 34 58 32 34 97 100 100 34 44 34 112 97 116 104 34 58 32 34 47 115 112 101 99 47 99 111 110 116 97 105 110 101 114 115 47 48 47 118 111 108 117 109 101 77 111 117 110 116 115 47 45 34 44 34 118 97 108 117 101 34 58 32 123 34 109 111 117 110 116 80 97 116 104 34 58 32 34 47 116 109 112 47 34 44 34 110 97 109 101 34 58 32 34 115 101 99 114 101 116 45 118 111 108 34 125 125 93],PatchType:*JSONPatch,AuditAnnotations:map[string]string{},}

The helm chart version that we are using is 0.1.2.

Hope you guys can lend us a hand.

[sai1621]

@Painyjames Thanks for testing the solution out.

Could you please share the output of below command to verify if the init container ran fine. kubectl describe pod webserver-5899d5548-md8zm | grep "Init Containers" -A 10

Note - Replace the webserver pod name with the correct one

The status should show something like

  secrets-init-container:
    Container ID:   docker://xxxxxyyyyzzzz
    Image:          docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.1
    Image ID:       docker-pullable://amazon/aws-secrets-manager-secret-sidecar@sha256:xxxxyyyyzzzz
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 17 Jul 2020 08:21:57 -0500

Now check the logs of the init container and the webserver pod with the below commands -

kubectl logs webserver-5899d5548-md8zm -c secrets-init-container kubectl logs webserver-5899d5548-md8zm

The output should be empty for the init container and display the secret value for the webserver pod.

Other steps to check -

• Verify if the OIDC provider exists in the IAM console under Identity providers section
• Verify if the IAM role webserver-secrets-role has the policy webserver-secrets-policy attached with the correct OIDC trust relationship

Please post your response here.

[victorarbuesmallada]

Apparently the pod is ok.

  secrets-init-container:
    Container ID:   docker://7b184e816e0d4f3270ba8c973b1f78fd508ac2f9e684922a5049f7c202b408ab
    Image:          docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.1
    Image ID:       docker-pullable://amazon/aws-secrets-manager-secret-sidecar@sha256:079028dde7738a622ecbf35ee6b345bba7a0a5f40584c4e97e21535ced0cc421
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 17 Jul 2020 10:53:48 +0100

but if we get the logs for the init container we receive the following:

kubectl logs webserver-7b56c5866-9sr5l  -c secrets-init-container
WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint
    status code: 400, request id: 7a767a2d-1a17-46a5-89d0-7ed93fb3120b

Might be this related to the certificates generated by the helm chart?

[sai1621]

You need to use the thumbprint of the OIDC provider, see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html

Try re-creating the OIDC provider and test the solution.

[victorarbuesmallada]

@sai1621 you were right, the OIDC thumbprint changed since the last time and we needed to recreate it. We are getting the secrets successfully now.