search

oracle / oci-native-ingress-controller

OCI Native Ingress Controller
https://blogs.oracle.com/cloud-infrastructure/post/oracle-cloud-native-ingress-controller-kubernetes
Universal Permissive License v1.0
19 stars 21 forks source link

Manage OCI Native Ingress Controller Load balancers #51

Closed naguaramaster closed 5 months ago

naguaramaster commented 6 months ago

Good afternoon greetings.

I have implemented the OCI Native Ingress Controller successfully, but it is observed that with each rule applied to the ingress resource a new backend set is created in the load balancer. When the backends set limit (16) is reached, how are the load balancers managed? knowing that above there is a DNS that points to the IP of a single load balancer. Is it a disadvantage of the service?

passarela commented 5 months ago

I have the same problem, I have applications with dozens of microservices, where each microservice is a path, so all together they total much more than 16 routes.

My initial idea is to use Nginx Ingress Controller, however the project for Oracle seems abandoned, I need integration with WAF and there is no sign for them to add it.

Is there any provision for exceeding the limit of 16 backendsets?

naguaramaster commented 5 months ago

I have the same problem, I have applications with dozens of microservices, where each microservice is a path, so all together they total much more than 16 routes.

My initial idea is to use Nginx Ingress Controller, however the project for Oracle seems abandoned, I need integration with WAF and there is no sign for them to add it.

Is there any provision for exceeding the limit of 16 backendsets?

Did you make any SR tickets to Oracle?

I opened an SR and the answer is that they are seeing this limitation with the development team, more than an error it is the administration of these balancers, it seems that the Native solution is not appropriate in terms of growth of services

passarela commented 5 months ago

I have the same problem, I have applications with dozens of microservices, where each microservice is a path, so all together they total much more than 16 routes. My initial idea is to use Nginx Ingress Controller, however the project for Oracle seems abandoned, I need integration with WAF and there is no sign for them to add it. Is there any provision for exceeding the limit of 16 backendsets?

Did you make any SR tickets to Oracle?

I opened an SR and the answer is that they are seeing this limitation with the development team, more than an error it is the administration of these balancers, it seems that the Native solution is not appropriate in terms of growth of services

I haven't opened SR yet, I'll be opening it this week. Unfortunately OCI native is not yet a large-scale solution.

naguaramaster commented 5 months ago

I have the same problem, I have applications with dozens of microservices, where each microservice is a path, so all together they total much more than 16 routes. My initial idea is to use Nginx Ingress Controller, however the project for Oracle seems abandoned, I need integration with WAF and there is no sign for them to add it. Is there any provision for exceeding the limit of 16 backendsets?

Did you make any SR tickets to Oracle? I opened an SR and the answer is that they are seeing this limitation with the development team, more than an error it is the administration of these balancers, it seems that the Native solution is not appropriate in terms of growth of services

I haven't opened SR yet, I'll be opening it this week. Unfortunately OCI native is not yet a large-scale solution.

At the moment we are implementing Nginx Ingress. There is an example in the documentation, apparently this Ingress does the task correctly.

antoniolago commented 5 months ago

The issue with ingress-nginx (at least in my case) is that ingress-nginx doesn't sync my cert-manager certificates with the load balancer, so we can only use ingress-nginx with a TCP Layer Load balancer (L4) and WAF only works at Application layer (L7). I'm considering using oci-native-ingress-controller (currently blocked by #56) so I can use WAF, but this 16 backends limits is very restraining.

naguaramaster commented 5 months ago

The issue with ingress-nginx (at least in my case) is that ingress-nginx doesn't sync my cert-manager certificates with the load balancer, so we can only use ingress-nginx with a TCP Layer Load balancer (L4) and WAF only works at Application layer (L7). I'm considering using oci-native-ingress-controller (currently blocked by #56) so I can use WAF, but this 16 backends limits is very restraining.

In my case, I have both options running in 2 different Clusters for testing and they are working. I opened an SR to Oracle and we demanded a PM to talk about the OCI Native Ingress Controller, since these limitations, whether technical or administrative, seem very strange to us and they do not have an adequate response for it. For now, with Ingress Nginx we need to replicate on multiple nodes, by default it only installs on a single node, which is also risky in case of reboot or node failure.

antoniolago commented 5 months ago

The issue with ingress-nginx (at least in my case) is that ingress-nginx doesn't sync my cert-manager certificates with the load balancer, so we can only use ingress-nginx with a TCP Layer Load balancer (L4) and WAF only works at Application layer (L7). I'm considering using oci-native-ingress-controller (currently blocked by #56) so I can use WAF, but this 16 backends limits is very restraining.

In my case, I have both options running in 2 different Clusters for testing and they are working. I opened an SR to Oracle and we demanded a PM to talk about the OCI Native Ingress Controller, since these limitations, whether technical or administrative, seem very strange to us and they do not have an adequate response for it. For now, with Ingress Nginx we need to replicate on multiple nodes, by default it only installs on a single node, which is also risky in case of reboot or node failure.

For ingress-nginx to deploy on multiple nodes you only have to set it to 

    controller:
      kind: DaemonSet

This being the helm value, but you can also change it in the deployment file.

We use ingress-nginx on oracle's load balancer and it works great, until we need things like ip source and WAF, which requires L7 load balancer, and having multiple domains and certificates would only work with oci-native-ingress's capability of syncing certificates. (or maybe I'm dead wrong and missing something)

naguaramaster commented 5 months ago

The issue with ingress-nginx (at least in my case) is that ingress-nginx doesn't sync my cert-manager certificates with the load balancer, so we can only use ingress-nginx with a TCP Layer Load balancer (L4) and WAF only works at Application layer (L7). I'm considering using oci-native-ingress-controller (currently blocked by #56) so I can use WAF, but this 16 backends limits is very restraining.

In my case, I have both options running in 2 different Clusters for testing and they are working. I opened an SR to Oracle and we demanded a PM to talk about the OCI Native Ingress Controller, since these limitations, whether technical or administrative, seem very strange to us and they do not have an adequate response for it. For now, with Ingress Nginx we need to replicate on multiple nodes, by default it only installs on a single node, which is also risky in case of reboot or node failure.

For ingress-nginx to deploy on multiple nodes you only have to set it to

    controller:
      kind: DaemonSet

This being the helm value, but you can also change it in the deployment file.

We use ingress-nginx on oracle's load balancer and it works great, until we need things like ip source and WAF, which requires L7 load balancer, and having multiple domains and certificates would only work with oci-native-ingress's capability of syncing certificates. (or maybe I'm dead wrong and missing something)

Oh, great! thank you!

I will continue to report here the progress I have with Oracle regarding OCI Native Ingress Controller.

antoniolago commented 5 months ago

Please do, I think this is an incredibly important piece of Oracle's Cloud Infrastructure and it's surprising to have such limitation.

Inbaraj-S commented 5 months ago

@antoniolago @naguaramaster You can reach out to CAMS team internally to increase the backendset limit. Defaults for LB are mentioned here : https://docs.oracle.com/en-us/iaas/Content/Balance/Concepts/balanceoverview.htm#LimitsResources

naguaramaster commented 5 months ago

@antoniolago @naguaramaster You can reach out to CAMS team internally to increase the backendset limit. Defaults for LB are mentioned here : https://docs.oracle.com/en-us/iaas/Content/Balance/Concepts/balanceoverview.htm#LimitsResources

Good morning, the response we got after creating the SR was successful. Our CXM spoke with a PM and the answer is that the service is not fully mature and they are still working on it in some aspects. Regarding the limits of backend sets, we were assured that they can be increased individually per Tenancy through an SR. So far it is the only issue that is observed and I think that with that answer we will follow the path of the OCI Native Ingress Controller, especially because they assure us of support, which does not happen if we adopt Nginx Ingress.