search

aws-samples / aws-secret-sidecar-injector

Kubernetes mutating webhook to fetch secrets from AWS Secrets Manager
MIT No Attribution
145 stars 35 forks source link

Init container isn't started for webserver + TLS handshake error in secret-inject #43

Closed kagarlickij closed 3 years ago

kagarlickij commented 3 years ago

Hi there! I'm following instructions but getting only webserver container in my webserver-78578795c6-6l2mx pod No init container is present there

In my secret-inject-87fd4b8bb-v6jvp pod logs I see http: TLS handshake error from 192.168.183.197:43904: remote error: tls: bad certificate exception on each webserver deploy attempt

Any ideas how to debug / fix it?

amit0701 commented 3 years ago

Hello,

That can typically occur when there is a cert mismatch between the admission controller and the mutating webhook configuration. I doubt if certs have expired (since they are valid for 3650 days). Is this a new clean-state deployment ? Verify that you dont have an existing secret named "secret-inject-tls" which was created as a part of previous Helm deployment.

kagarlickij commented 3 years ago

I've just completed full k8s redeploy and still getting exactly the same. Worse mentioning that other sample deploy that is using IAM OIDC, role and policy to reach S3 via Amazon EKS Pod Identity Webhook works just fine.

I also tried to upd trust policy's condition to be:

"StringLike": {
    "oidc.eks.eu-west-1.amazonaws.com/id/5D31CCC00A12B76DB39BBBD6E038775E:sub": "system:serviceaccount:default:*"
}

..instead of:

"StringEquals": {
    "oidc.eks.eu-west-1.amazonaws.com/id/5D31CCC00A12B76DB39BBBD6E038775E:sub": "system:serviceaccount:default:webserver-service-account",
    "oidc.eks.eu-west-1.amazonaws.com/id/5D31CCC00A12B76DB39BBBD6E038775E:aud": "sts.amazonaws.com"
}

..but no luck :(

I have k8s v1.19, nodePool v1.19.6-20210302, CNI v1.7.9-eksbuild.1 if it matters

shashankbansal6 commented 3 years ago

I have the same issue. 

2021/03/11 00:13:02 http: TLS handshake error from ********: remote error: tls: bad certificate
2021/03/11 00:13:02 http: TLS handshake error from ********: remote error: tls: bad certificate

K8s details:

kubectl version --short                                                                                                                                                                
Client Version: v1.19.3
Server Version: v1.19.6-eks-49a6c0
jicowan commented 3 years ago

@shashankbansal6 and @kagarlickij we're testing this against 1.19 now. Perhaps something changed in that release.

themattkeating commented 3 years ago

I'm now also having this issue after upgrading to 1.19. Note: I'm running the multi-secret branch, images in my own ECR.

>  ❯ kubectl version --short                                                                                                                                        

Client Version: v1.19.3
Server Version: v1.19.6-eks-49a6c0
jicowan commented 3 years ago

I believe we need to add a SAN to the certificate.

amit0701 commented 3 years ago

@kagarlickij @shashankbansal6 This should be fixed.

themattkeating commented 3 years ago

@amit0701 @jicowan Please can you confirm where you added the SAN? I dont see any commits to the repo and I'm using the multi-secret branch.

jicowan commented 3 years ago

We're not actively maintaining that branch @themattkeating. You could ask @dnascimento if he has plans to update the certificate.

themattkeating commented 3 years ago

Unfortunately the master branch only supports a single secret, which is almost useless in the real world (it kinda forces the use of the multi-secret branch).

@jicowan If you could explain how the certificate was changed, Im happy to do a PR for this multi-secret branch. I just wanted to know more about what the fix was.

jicowan commented 3 years ago

The changes are in the gh-pages branch: https://github.com/aws-samples/aws-secret-sidecar-injector/commits/gh-pages