search

emissary-ingress / emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy
https://www.getambassador.io
Apache License 2.0
4.34k stars 681 forks source link

Is it possible to change ports on apiext deployment? #4196

Open EvertonSA opened 2 years ago

EvertonSA commented 2 years ago

https://github.com/emissary-ingress/emissary/blob/5f01e923a297182a08217b7651bfc1a77735391f/cmd/apiext/main.go#L54

context:

we are migration ambassador edge stack from 1.14 to 2.2 and this port is blocking the webhooks in a EKS cluster with calico. I need to use hostNetwork for my webhooks due to calico and I need to chose a different port for the setup. If I use raw as is deployments from this tutorial: https://www.getambassador.io/docs/edge-stack/latest/topics/install/upgrade/yaml/edge-stack-1.14/edge-stack-2.2/ I get the following error:

kubectl -n ambassador logs -f aes-9598676c-56cv9

E0417 20:41:05.376019       1 reflector.go:138] pkg/kates/client.go:439: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: conversion webhook for getambassador.io/v2, Kind=Mapping failed: Post "https://emissary-apiext.emissary-system.svc:443/webhooks/crd-convert?timeout=30s": Address is not allowed
jbdacruz commented 2 years ago

https://github.com/emissary-ingress/emissary/blob/5f01e923a297182a08217b7651bfc1a77735391f/cmd/apiext/main.go#L54

context:

we are migration ambassador edge stack from 1.14 to 2.2 and this port is blocking the webhooks in a EKS cluster with calico. I need to use hostNetwork for my webhooks due to calico and I need to chose a different port for the setup. If I use raw as is deployments from this tutorial: https://www.getambassador.io/docs/edge-stack/latest/topics/install/upgrade/yaml/edge-stack-1.14/edge-stack-2.2/ I get the following error:

kubectl -n ambassador logs -f aes-9598676c-56cv9

E0417 20:41:05.376019       1 reflector.go:138] pkg/kates/client.go:439: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: conversion webhook for getambassador.io/v2, Kind=Mapping failed: Post "https://emissary-apiext.emissary-system.svc:443/webhooks/crd-convert?timeout=30s": Address is not allowed

I'm hitting this right now. Did you figure out how to solve this?

EvertonSA commented 2 years ago

sorry @jbdacruz , I was not able to solve it

cindymullins-dw commented 2 years ago

Kubernetes is what talks to apiext, and it expects to use standard HTTPS. So… you could tweak the service definition a bit, maybe, but it's unclear from our perspective what the point would really be. Is it possible the firewall in EKS is blocking things?

mpluhar commented 2 years ago

Same issue here in EKS just following the examples in the emissary-ingress documentation when using Cilium in overlay networking mode. Guess the issue is not directly related to emissary, but an issue with EKS network configuration.

matdehaast commented 2 years ago

@cindymullins-dw the configuration is required so that when hostNetwork: true is chosen that we could choose the ports to use. Currently the ports 8080, 8443 could be taken up by another item on the host. By being able to configure it, we could set it ourselves if there are conflicts.

I'm happy to create a PR for this. I'm just not sure how best to set the values? ENV variables seem the easiest

aspexdaniel commented 1 year ago

hit the same issue with EKS cluster & Calico. The main reason is mentioned at Calico's website: https://projectcalico.docs.tigera.io/getting-started/kubernetes/managed-public-cloud/eks

Note: Calico networking cannot currently be installed on the EKS control plane nodes. As a result the control plane nodes will not be able to initiate network connections to Calico pods. (This is a general limitation of EKS’s custom networking support, not specific to Calico.) As a workaround, trusted pods that require control plane nodes to connect to them, such as those implementing admission controller webhooks, can include hostNetwork:true in their pod spec. See the Kubernetes API pod spec definition for more information on this setting.

Basically the only thing we can do is to use hostNetwork:true, then apiext by default will only use 8080, 8443 which obviously is not ideal. It should be configurable.

For reference, cert-manager has a nice documentation exactly about this https://cert-manager.io/docs/installation/compatibility/ and they provide a nice configurable value webhook.securePort in their helm chart.

cindymullins-dw commented 1 year ago

@matdehaast and @aspexdaniel thanks for that context. I'll ask about this at our next Emissary Contributors meeting in Jan. You're also welcome to attend. If this is something we can do and you're still interested in contributing it'd be helpful if you could join our #emissary-dev channel in our Slack.

gutozntt commented 1 year ago

@EvertonSA @aspexdaniel @jbdacruz hey guys, were you able to work around this issue? We had to update ambassador to a new version due to a cluster upgrade and ended up facing this same issue.

cindymullins-dw commented 1 year ago

@matdehaast if this is still posing problems for you, are you still interested in contributing to this change?

EvertonSA commented 1 year ago

I have not figure out @gutozntt, I left the company, but afaik this problem still exists on aws. on gcp seems to be all good

jmtsi commented 1 year ago

We encountered this issue today. We tried to solve it by configuring the deployment to use hostnetworking and then configuring the Emissary CRDs to use different port for the conversion webhook. The default configuration for every CRD looks like this:

    strategy: Webhook
    webhook:
      clientConfig:
        service:
          name: emissary-apiext
          namespace: emissary-system

According to Kubernetes Documentation, we should be able to hardcode the port by simply configuring it like this:

    strategy: Webhook
    webhook:
      clientConfig:
        service:
          name: emissary-apiext
          namespace: emissary-system
          port: 9443

But after creating the CRDs with this modified config, describing the CRD shows that the hardcoded port is not used:

        Service:
          Name:       emissary-apiext
          Namespace:  emissary-system
          Path:       /webhooks/crd-convert
          Port:       443

So for me this looks like a bug. Something seems to override the config during apply, which shouldn't happen.

jmtsi commented 1 year ago

Seems like the port will be overwritten by emissary-apiext. The 443 port is hardcoded here: https://github.com/emissary-ingress/emissary/blob/19f9e0069e38beb87320e465cc8377335c15b2f9/cmd/apiext/inject.go#LL52C27-L52C27