search

Kong / kubernetes-ingress-controller

:gorilla: Kong for Kubernetes: The official Ingress Controller for Kubernetes.
https://docs.konghq.com/kubernetes-ingress-controller/
Apache License 2.0
2.2k stars 590 forks source link

Support single controller deployments #702

Closed hbagdi closed 1 year ago

hbagdi commented 4 years ago

Problem Statement

Currently the ingress controller is only capable of deploying configuration to one Kong Gateway at a time, the consequence of which is that to horizontally scale the data-plane in DBLESS mode you must create an ingress controller for each gateway which has some performance costs and wastes associated with it. The purpose of this issue is to make it possible to configure the KIC with multiple Gateways so that they all receive the same configuration updates.

Proposed Solution

Additional information

Acceptance Criteria

seh commented 4 years ago

We are definitely interested in this capability. If you need help, please let me know.

marcoam commented 4 years ago

+1

shaneutt commented 3 years ago

Plan:

  1. proxy containers will now have their own pod in the Kong Ingress Controller
  2. controller and proxies will have mTLS now that they will communicate over the container network and not over local host
  3. service discovery
  4. reconciliation needs to operate on N proxies now
  5. include NetworkPolicy resources with KIC deployment which restrict access to the Kong Admin APIs now that they'll be exposed over the container network.

Notes:

shaneutt commented 3 years ago

After discussing further with @mflendrich we're going to take some time to do some further investigation on the problem domain here (as opposed to going right into implementation as currently written) as we feel that there's some potential for large gains in rethinking our approach to this crossroads in how we deploy and manage the proxy.

shaneutt commented 3 years ago

The team has decided that we'll hold on this issue for now as we try to make some architectural changes to the KIC which will ultimately make implementing this easier.

swapnilpotnis commented 3 years ago

Plan:

  1. proxy containers will now have their own pod in the Kong Ingress Controller
  2. controller and proxies will have mTLS now that they will communicate over the container network and not over local host
  3. service discovery
  4. reconciliation needs to operate on N proxies now
  5. include NetworkPolicy resources with KIC deployment which restrict access to the Kong Admin APIs now that they'll be exposed over the container network.

Notes:

  • mTLS is on Kong side
  • need to update kong/kong-operator, kong/charts

So does this mean, in future, kong would be able to support mTLS communication between kong <-> service?? If so, then could you please let us know if https://docs.konghq.com/kubernetes-ingress-controller/1.1.x/guides/upstream-mtls/ is something different than what is being implemented??

We basically have a use-case where we have an outside service communicating with kong running over AKS. Now we wish to establish mTLS between the two. As we have installed kong from helm charts, we were not able to accomodate the variables as mentioned in the above given link. Hence wanted to know if it was possible with the current kong release or is the mTLS support by kong yet to be implemented?

shaneutt commented 3 years ago

So does this mean, in future, kong would be able to support mTLS communication between kong <-> service??

In this particular case:

This covers mTLS between the Kubernete Ingress Controller Pod and access to the Admin API in the Kong Proxy Pod. Note that historically the containers have been in a single pod together and used localhost connectivity so mTLS was not enabled.

This does not pertain to any other services (e.g. the services being proxied to) for this scope.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

shaneutt commented 3 years ago

Several weeks ago we started some work on this but it was not finished due to a desire by the maintainers to wait until KIC 2.0 which will bring a major re-architecture that would better support this feature.

We'll consider this blocked on the KIC 2.0 milestone for now and revisit once that is completed.

shaneutt commented 2 years ago

As stated before we're still considering the potential synergy with a future KIC operator, but we're also now considering the synergy of this improvement with the advent of Gateway support. The relevant KEP is being re-opened and modernized to reflect recent developments: https://github.com/Kong/kubernetes-ingress-controller/pull/2256

shaneutt commented 2 years ago

Relevant to the mTLS portion of this, flags have been added as part of #1958 to provide client certs for mTLS with the Kong Gateway. The remaining portion of the work which relates to being able to support multiple Gateways with a single KIC is still covered by this issue and the KEP https://github.com/Kong/kubernetes-ingress-controller/blob/main/keps/0001-single-controller-multi-proxy.md.

mflendrich commented 2 years ago

One option to consider in this spike as raised by @rainest: use alternative control plane solutions to achieve this problem.

ludovic-pourrat commented 2 years ago

Hi,

Within this KEP https://github.com/Kong/kubernetes-ingress-controller/blob/main/keps/0001-single-controller-multi-proxy.md, will the support of the hybrid mode with the dbless feature is an option ?

shaneutt commented 2 years ago

Hi,

Within this KEP https://github.com/Kong/kubernetes-ingress-controller/blob/main/keps/0001-single-controller-multi-proxy.md, will the support of the hybrid mode with the dbless feature is an option ?

With hybrid mode you don't generally need more than one ingress controller (if you're using the ingress controller at all), so currently hybrid mode isn't really a prominent part of the direction so far. This feature would in fact represent an alternative to hybrid mode, at least from the perspective of providing horizontal scaling options for the dataplane (kong gateway).

Was there something specific you needed or were hoping for?

ludovic-pourrat commented 2 years ago

@shaneutt thanks for you reply, and what we are hoping for is to run the hybrid mode with the dbless feature (with one ingress controller instance, one or two control planes and a brunch of data planes), which is not supported at the moment. Secondly from an availability point of view, the hybrid mode seem's more capable to handle restart and failures of the data plane, where in the current dbless implementation it takes more time for the data plane to resurface with a valid setup. In a sense this was about taking benefit from the current hardening of the hybrid mode but with the dbless feature and with better resiliency of the control plane if this kep where taking the hybrid mode as an option.

shaneutt commented 2 years ago

@shaneutt thanks for you reply, and what we are hoping for is to run the hybrid mode with the dbless feature (with one ingress controller instance, one or two control planes and a brunch of data planes), which is not supported at the moment. Secondly from an availability point of view, the hybrid mode seem's more capable to handle restart and failures of the data plane, where in the current dbless implementation it takes more time for the data plane to resurface with a valid setup. In a sense this was about taking benefit from the current hardening of the hybrid mode but with the dbless feature and with better resiliency of the control plane if this kep where taking the hybrid mode as an option.

I see. This issue isn't related to hybrid mode so it would seem we should take this conversation elsewhere. Would you please put in a feature request for the specific requirements you have as a separate issue?

ludovic-pourrat commented 2 years ago

@shaneutt sure thx !

mflendrich commented 1 year ago

May be implemented by https://github.com/Kong/kubernetes-ingress-controller/issues/3209

pmalek commented 1 year ago

This is currently being worked on.

3268 aims to enable rendering and sending configs for multiple Kong Gateways: this will be done via the same flag that's in place today --kong-admin-url (by passing multiple, comma separated values).

When that's done then we'll work on making this not be bound to controller's Pod's lifetime, specifically to use a label selector for Kong's Admin API Service.

If all that gets released we'll be able to then incorporate those changes into helm chart so that users can utilize that deployment method to deploy single controller with multiple Gateways.

mflendrich commented 1 year ago

The remaining work here is to complete all the remaining checkboxes in the AC.

czeslavo commented 1 year ago

We still have

For any number of data-planes, the KIC should be able to be configured with separate mTLS credentials for each

in ACs which didn't have its tracking issue. I created one, but I'm not sure how would that work having gateways discovered dynamically via a headless service. 🤔 Did we try to design this anywhere?

pmalek commented 1 year ago

We still have

For any number of data-planes, the KIC should be able to be configured with separate mTLS credentials for each

in ACs which didn't have its tracking issue. I created one, but I'm not sure how would that work having gateways discovered dynamically via a headless service. 🤔 Did we try to design this anywhere?

For posterity: as agreed on sync meeting #3603 will track this effort but it won't be land in v2.9.

mflendrich commented 1 year ago

The core has been implemented and will ship in KIC 2.9 and the Helm chart 2.17.

There are a few open items left:

Now that we have only those details to polish, I'm closing this huge issue in favor of the more targeted ones above.