[eks] [bug]: eks api unable to call webhook's when using the custom cni networking

[bseenu]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request eks api unable to call webhook's when using the custom cni networking

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? I am trying to create clusterissuer object of cert-manager i.e https://cert-manager.io/docs/

sboga@C02SW14YG8WL ~ % kubectl api-resources|grep -i clusterissuers clusterissuers cert-manager.io false ClusterIssuer

And creating this object involves api contacting the webhook api svc for validation which is failing

Logs from kubectl apply -f /tmp/file -v=9

I0106 20:24:00.361568 63216 request.go:1097] Request Body: {"apiVersion":"cert-manager.io/v1","kind":"ClusterIssuer","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"cert-manager.io/v1\",\"kind\":\"ClusterIssuer\",\"metadata\":{\"annotations\":{},\"name\":\"letsencrypt-prod1\"},\"spec\":{\"acme\":{\"email\":\"hostmaster@roku.com\",\"privateKeySecretRef\":{\"name\":\"letsencrypt-prod\"},\"server\":\"https://acme-v02.api.letsencrypt.org/directory\",\"solvers\":[{\"dns01\":{\"route53\":{\"hostedZoneID\":\"Z27VCBU17XD4X4\",\"region\":\"us-east-1\",\"role\":\"arn:aws:iam::088414020449:role/dns-rokulabs-net-manager-role\"}},\"selector\":{\"dnsZones\":[\"rokulabs.net\"]}},{\"dns01\":{\"route53\":{\"hostedZoneID\":\"Z3S2XVQIXW67TY\",\"region\":\"us-east-1\",\"role\":\"arn:aws:iam::824163854805:role/dns-msc-dev-roku-com-manager-role\"}},\"selector\":{\"dnsZones\":[\"msc-dev.roku.com\"]}}]}}}\n" },"name":"letsencrypt-prod1"},"spec":{"acme":{"email":"hostmaster@roku.com","privateKeySecretRef":{"name":"letsencrypt-prod"},"server":"https://acme-v02.api.letsencrypt.org/directory","solvers":[{"dns01":{"route53":{"hostedZoneID":"Z27VCBU17XD4X4","region":"us-east-1","role":"arn:aws:iam::088414020449:role/dns-rokulabs-net-manager-role"}},"selector":{"dnsZones":["rokulabs.net"]}},{"dns01":{"route53":{"hostedZoneID":"Z3S2XVQIXW67TY","region":"us-east-1","role":"arn:aws:iam::824163854805:role/dns-msc-dev-roku-com-manager-role"}},"selector":{"dnsZones":["msc-dev.roku.com "]}}]}}} I0106 20:24:00.361765 63216 round_trippers.go:423] curl -k -v -XPOST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.19.0 (darwin/amd64) kubernetes/e199641" 'https://3CAC058C6DE86A596295535EC5B4E9C7.gr7.us-west-2.eks.amazonaws.com/apis/cert-manager.io/v1/clusterissuers?fieldManager=kubectl-client-side-apply ' I0106 20:24:00.408237 63216 round_trippers.go:443] POST https://3CAC058C6DE86A596295535EC5B4E9C7.gr7.us-west-2.eks.amazonaws.com/apis/cert-manager.io/v1/clusterissuers?fieldManager=kubectl-client-side-apply 500 Internal Server Error in 46 milliseconds I0106 20:24:00.408276 63216 round_trippers.go:449] Response Headers: I0106 20:24:00.408283 63216 round_trippers.go:452] Content-Type: application/json I0106 20:24:00.408290 63216 round_trippers.go:452] Content-Length: 479 I0106 20:24:00.408296 63216 round_trippers.go:452] Date: Thu, 07 Jan 2021 04:24:00 GMT I0106 20:24:00.408302 63216 round_trippers.go:452] Audit-Id: ec534821-9753-4377-8f7b-7b8ab014243e I0106 20:24:00.408308 63216 round_trippers.go:452] Cache-Control: no-cache, private I0106 20:24:00.408355 63216 request.go:1097] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed"}]},"code":500} I0106 20:24:00.408706 63216 helpers.go:216] server response object: [{ "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed", "reason": "InternalError", "details": { "causes": [ { "message": "failed calling webhook \"webhook.cert-manager.io\": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed" } ] }, "code": 500 }] F0106 20:24:00.408756 63216 helpers.go:115] Error from server (InternalError): Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: Address is not allowed Region: us-west-2 Cluster Name: Instance ID(s):

Are you currently working around this issue? Did not find any

[fabianpeterg]

the cert-manager docs has a workaround for this: https://cert-manager.io/docs/installation/compatibility/ it is pretty bad nevertheless that EKS does not really work well with 3rd party CNIs..

[josephprem]

We have a bunch of webhooks in our deployments. Our current setup involves an internal nginix ingress controller and exposing the webhooks via ingress. Obviously, nginx is on host network