Unable to add ingress successfully

[grk-pancham]

Describe the bug I followed the steps outlined here (https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/) to install nginx ingress controller in Kubernetes v1.17. I get error when i add the ingress resource. Please advice.

To Reproduce Install ingress controller as outline above install app, services and ingress with this command kubectl create -f cafe.yaml kubectl create -f ingress.yaml

cafe.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tea
spec:
  selector:
    matchLabels:
      app: tea
  replicas: 2
  template:
    metadata:
      labels:
        app: tea
    spec:
      containers:
      - name: tea
        image: nginxdemos/hello
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
  labels:
    app: tea
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: tea
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coffee
spec:
  selector:
    matchLabels:
      app: coffee
  replicas: 2
  template:
    metadata:
      labels:
        app: coffee
    spec:
      containers:
      - name: coffee
        image: nginxdemos/hello
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: coffee-svc
  labels:
    app: coffee
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: coffee

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
spec:
  rules:
  - host: mydomain.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

Expected behavior I was expecting ingress resource to be added successfully but i see errors in the ingress controller.

Errors:

W0115 21:50:37.726827       1 controller.go:1527] Error retrieving endpoints for the service tea-svc: No endpoints for target port 80 in service tea-svc
W0115 21:50:37.726861       1 controller.go:1527] Error retrieving endpoints for the service coffee-svc: No endpoints for target port 80 in service coffee-svc
2020/01/15 21:50:37 [notice] 80#80: signal process started
 -  []
 -  []
I0115 21:50:41.787708       1 event.go:255] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cafe-ingress", UID:"ce1a0374-53fe-4209-9c6c-aa56ab2abe68", APIVersion:"extensions/v1beta1", ResourceVersion:"23814", FieldPath:""}): type: 'Warning' reason: 'AddedOrUpdatedWithError' Configuration for default/cafe-ingress was added or updated, but not applied: Error reloading NGINX for default/cafe-ingress: could not get newest config version: could not get expected version: 8
2020/01/15 21:50:41 [notice] 83#83: signal process started
E0115 21:50:45.825729       1 controller.go:407] Error updating endpoints for [default/cafe-ingress]: Error reloading NGINX when updating endpoints: could not get newest config version: could not get expected version: 9

Your environment Kubernetes v1.17.0

• Version of the Ingress Controller - release version or a specific commit edge
• Version of Kubernetes v1.17.0
• Kubernetes platform (e.g. Mini-kube or GCP) Kubeadm
• Using NGINX or NGINX Plus NGINX

Additional context Add any other context about the problem here. Any log files you want to share.

[Rulox]

Hey @grk-kulkarni I am sorry to hear this is not working for you. In order to see what's happening I am going to need you to provide the following information:

1. Have you done any change to the manifests or ConfigMaps from the cafe example? (eg you are using ConfigMap keys or annotations in your ingress resources)
2. Could you possibly deploy the Ingress Controller with the the -v cli argument set to 3? -v 3 (more info on cli arguments here) and paste the logs again with that change?
3. Also, could you paste the outcome of the following commands (in the namespace where you are deploying the ingress resources)?: a. kubectl get pods b. kubectl get svc c. kubectl get endpoints

Thanks!

[grk-pancham]

I have a known issue on my K8 cluster that could be the cause of this issue too. Docker containers in my kubernetes cluster are not able to reach out. The firewall is blocking the egress access for docker container and I am looking to update the firewall rules. I am guessing ingress controller pod is failing to connect to kube api server to get service endpoints. i will try to add the logs as requested soon.

[grk-pancham]

kubectl get svc --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default coffee-svc ClusterIP 10.99.250.85 80/TCP 13s default kubernetes ClusterIP 10.96.0.1 443/TCP 29h default tea-svc ClusterIP 10.104.62.77 5432/TCP 14s kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 29h

kubectl get endpoints NAME ENDPOINTS AGE coffee-svc 192.168.70.33:80,192.168.70.34:80 42s kubernetes 10.192.50.212:8443 29h tea-svc 192.168.70.32:5432,192.168.70.35:5432 42s

kubectl get pods NAME READY STATUS RESTARTS AGE coffee-67dfbbb5c5-bmv9q 1/1 Running 0 71s coffee-67dfbbb5c5-wpcj4 1/1 Running 0 71s tea-5c8d998757-4cwwm 1/1 Running 0 71s tea-5c8d998757-9s27g 1/1 Running 0 72s

[Rulox]

That could be a reason, however, I would expect more errors (The IC needs to connect to the k8s API to retrieve things like its ConfigMap, without that, the IC would fail with an error).

Thanks for the response, it seems everything is fine. It could be nice if you can share also the logs with the log level v=3 to see if there's something else happening there.

I'll wait for them now, thanks!

[grk-pancham]

Hi Rulox, I added new nifi service and ingress for it. attached is the debug log (with -v=3). I see two errors that is causing the issue. not sure what the resolution is. please advice.

I0117 17:43:14.033040 1 controller.go:781] Adding or Updating Ingress: default/nifi-ingress I0117 17:43:14.033057 1 controller.go:1838] Error getting endpoints for service nifi port {0 30100 }: No port {0 30100 } in service nifi W0117 17:43:14.033064 1 controller.go:1527] Error retrieving endpoints for the service nifi: No port {0 30100 } in service nifi

I0117 17:43:14.067351 1 verify.go:70] Unable to fetch version: error getting client: Get http://config-version/configVersion: dial unix /var/lib/nginx/nginx-config-version.sock: connect: no such file or directory

kc get endpoints kubernetes 10.192.50.198:8443,10.192.50.235:8443 8d nifi 192.168.214.193:8080 7d2h

kc get po nifi-5fdd8cf994-tdgxd 1/1 Running 0 21h

kc get svc kubernetes ClusterIP 10.96.0.1 443/TCP 8d nifi NodePort 10.96.220.159 8080:30100/TCP 7d2h

ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nifi-ingress spec: rules:

• host: mydomain.com http: paths:
  • path: /nifi backend: serviceName: nifi servicePort: 30100

debug.log: attached nginx.log

[grk-pancham]

Hi Rulox, After updating the servicePort from 30100 to 8080 in ingress.yaml, I do not get the first error. but the nginx controller still does not redirect me to the service page and i get 404 Not Found from nginx controller. I see this error in the logs

2020/01/17 19:34:15 [error] 85#85: *2944 open() "/usr/share/nginx/html/nifi" failed (2: No such file or directory), client: 10.192.50.198, server: localhost, request: "GET /nifi HTTP/1.1", host: "10.192.50.155:8080"

I was expecting controller to redirect to service page but looks like it is trying to open nifi site location at /usr/share/nginx/html/nifi. how to you make ingress controller to redirect to the nifi service and not open site locally.

fyi. i have updated the default nginx.conf and default.conf @ /etc/nginx on both controller pods since i was getting 403 forbidden with the default config that came with the image. I do not think the error is caused by this change though.

this is the nifi ingress kc describe ingress nifi-ingress Name: nifi-ingress Namespace: default Address: Default backend: default-http-backend:80 () Rules: Host Path Backends

---

mydomain /nifi nifi:8080 (192.168.214.193:8080) Annotations: Events: Type Reason Age From Message

---

Warning AddedOrUpdatedWithError 19m nginx-ingress-controller Configuration for default/nifi-ingress was added or updated, but not applied: Error reloading NGINX for default/nifi-ingress: could not get newest config version: could not get expected version: 11 Warning AddedOrUpdatedWithError 19m nginx-ingress-controller Configuration for default/nifi-ingress was added or updated, but not applied: Error reloading NGINX for default/nifi-ingress: could not get newest config version: could not get expected version: 11

[grk-pancham]

Hi Rulox, when i describe on nifi-ingress, i see warning which are causing the redirect to service failing. Please advice on the resolution.

I0117 20:23:05.499034 1 event.go:255] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nifi-ingress", UID:"471a34d3-595b-452e-a2b9-0aec0682a65e", APIVersion:"extensions/v1beta1", ResourceVersion:"2000191", FieldPath:""}): type: 'Warning' reason: 'AddedOrUpdatedWithError' Configuration for default/nifi-ingress was added or updated, but not applied: Error reloading NGINX for default/nifi-ingress: could not get newest config version: could not get expected version: 15

I see this error in the log which happens during reloading of nifi-ingress. I0117 20:23:04.970049 1 verify.go:70] Unable to fetch version: error getting client: Get http://config-version/configVersion: dial unix /var/lib/nginx/nginx-config-version.sock: connect: no such file or directory.

I think the above is causing the redirect to fail. please advice.

[grk-pancham]

I added this line to the /etc/nginx/nginx.conf and the error during ingress creation seems to have gone. Now the ingress is correctly updated but the redirect does not happen and do not get the service page yet. I get 404 not found from controller. I see this error in logs. why does the redirection does not happen.

2020/01/17 21:17:13 [error] 119#119: *5425 "/usr/share/nginx/html/nifi/index.html" is not found (2: No such file or directory), client: 10.192.50.251, server: localhost, request: "GET /nifi/ HTTP/1.1", host: "mydomain.com"

include /etc/nginx/config-version.conf;

kc describe nifi-ingress `Name: nifi-ingress Namespace: default Address: Default backend: default-http-backend:80 () Rules: Host Path Backends

---

mydomain.com /nifi nifi:8080 (192.168.214.193:8080) Annotations: Events: Type Reason Age From Message

---

Normal AddedOrUpdated 116s nginx-ingress-controller Configuration for default/nifi-ingress was added or updated Normal AddedOrUpdated 116s nginx-ingress-controller Configuration for default/nifi-ingress was added or updated`

[Rulox]

Hey @grk-kulkarni thanks for the detailed info, let's see if we can get this working,

Ok a few things:

I was expecting controller to redirect to service page but looks like it is trying to open nifi site location at /usr/share/nginx/html/nifi. how to you make ingress controller to redirect to the nifi service and not open site locally.

That's not how it works. NGINX will redirect to the configured service by default, so there is something wrong in your configuration that prevents this, we need to find out what. I think the reason is this:

upstream default-nifi-ingress-gtwy-udev.gateway-imd.boozallencsn.com-nifi-30100 {
    zone default-nifi-ingress-gtwy-udev.gateway-imd.boozallencsn.com-nifi-30100 256k;
    random two least_conn;

    server 127.0.0.1:8181 max_fails=1 fail_timeout=10s max_conns=0;

}

Because the IC can't get the endpoints for your service, it is using 127.0.0.1. So I think this is related to the first problem you had from the beginning.

fyi. i have updated the default nginx.conf and default.conf @ /etc/nginx on both controller pods since i was getting 403 forbidden with the default config that came with the image. I do not think the error is caused by this change though.

You can't update the config manually, this will cause failures. The Ingress Controller is in charge of the updates of the NGINX configs, if you do a manual change, 2 things will happen:

1. The changes will be overwritten by the Ingress Controller when any resource that is watching is changed.
2. You can end up with an illegal NGINX config (error ... could not get newest config version: could not get expected version: X)

When you get this error could not get newest config version: could not get expected version: X that means the NGINX config is not updated, no matter what you do, the IC won't pick up any changes unless the conf is back to normal (this could happen if you have an invalid config of NGINX)

Can you provide the following:

• Manifest for the ingress resource of your nifi svc
• Manifests of your Service and deployment (if any) of your nifi service
• The ConfigMap for the Ingress Controller?

Also one question: did you follow the cafe complete example step by a step in a fresh kubernetes cluster? That example is the one we always use, and it's used in our integration tests, so we can assume it works, but it seems to me there's something in your cluster that might be preventing the IC to work correctly. Don't worry we'll find it!

Thanks! I'll wait for those last manifests

[grk-pancham]

Hi Rulox,

Attached are the manifests. nifi.txt

I will redeploy IC with default config and see if that works.

I left the cafe example and started testing IC with nifi service. I will give it a try too.

Thanks for looking into this issue.

[grk-pancham]

Hi Rulox,

I redeployed IC with default config (that comes with the image). I see that ingress is updated without any errors.

I0121 02:35:42.484102 1 event.go:255] Event(v1.ObjectReference{Kind:"Secret", Namespace:"nginx-ingress", Name:"default-server-secret", UID:"d33ea16e-034a-4c4a-9b19-638da37472eb", APIVersion:"v1", ResourceVersion:"2732494", FieldPath:""}): type: 'Normal' reason: 'Updated' the special Secret nginx-ingress/default-server-secret was updated

I see this config updated by IC.

`upstream default-nifi-ingress-mydomain-8080 { zone default-nifi-ingress-mydomain-nifi-8080 256k; random two least_conn;

server 192.168.214.193:8080 max_fails=1 fail_timeout=10s max_conns=0;

}`

But I get 403 Forbidden error from IC when I try to open the service page. I see below errors in IC log

10.192.50.155 - - [21/Jan/2020:02:35:45 +0000] "GET / HTTP/1.1" 403 153 "-" "ELB-HealthChecker/2.0" "-" 2020/01/21 02:35:48 [error] 35#35: *6 access forbidden by rule, client: 10.192.50.155, server: , request: "GET / HTTP/1.1", host: "10.192.50.155:8080"

I think the IC config is missing something

Please advice what I am missing.

[Rulox]

Great! That makes much more sense @grk-kulkarni, that config seems correct (if that IP is the ip from you pod svc of course).

I can see that request is from an ELB Health Check? How do you have configured your healthcheck in aws ELB? (ip, port, path, etc).

Have you tried to do a manual request? Via curl or with your browser?

[grk-pancham]

Hi Rulox,

the target type on AWS ALB is instance, port is 8080, health check port is 8080, path is / and success code 403. success code on health check is 403 since i get 403 forbidden if i hit the IC. the health check fails if i change success code to 200.

from browser and curl , i get same response which is 403 forbidden page from IC.

what should be the response i should get from IC if I do curl on port 8080 where IC is listening to.

[Rulox]

Ok so, a couple of things:

We provide a way to configure an external HealthCheck endpoint. Please check this docs.

You can activate the healthcheck endpoint by setting -health-status to true in the cli-arguments and then, if needed, specify the healthcheck uri (default is /nginx-health). That should respond with a 200 in the port that you have specified (80 or 443) for your IC service to run. This is the correct way.

About the 403, that is weird. In order to check where that 403 is coming from, apart from the log-level, there is one way to run NGINX as debug mode, that will provide a lot of information. Check this docs. Then your IC logs will provide a detailed info in every request, I think that will tell us what's happening.

Let me know, thanks!

[grk-pancham]

Hi Rulox,

I redeployed IC with debug logging and health check enabled. I still get 403 Forbidden on the health check endpoint (/nginx-health) and on nifi service too. I am not sure what is missing here. why does it say "Access forbidden by rule". Attached is the IC debug log after starting.

I se health check status is updated in nginx.conf location /nginx-health { default_type text/plain; return 200 "healthy\n"; } nginx.log

[Rulox]

Hmmmm, everything looks correct there.

Ah! I think I know the issue, from your previous comment:

the target type on AWS ALB is instance, port is 8080, health check port is 8080, ...

This is wrong, the Ingress Controller ports for external incoming traffic are 80 or 443 (by default). 8080 is only for internal status checks. If you see the config:

    # stub_status
    server {
        listen 8080;

        allow 127.0.0.1;
        deny all;

        location /stub_status {
            stub_status;
        }
    }

The deny all directive prevents any request (403) except the ones from localhost (note the aallow 127.0.0.1;`)

The server config for the status location (and also ingress resources) is like this:

    server {
        listen 80 default_server;
        listen 443 ssl default_server;
...
        location /nginx-health {
            default_type text/plain;
            return 200 "healthy\n";
        }

    }

Note how NGINX is listening on 80 an 443. Can you please try to use ports 80 or 443 instead and let me know?

Cheers

[grk-pancham]

Hi Rulox,

Good catch. is there way to change the default port from 80 to some other port. We have a forward from port 80 to 8080 as part of our firewall rules. Since 8080 is used by stub_status, I can use some other port other than 80 or 8080 for default port.

I changed the containerPort and hostPort to 8080 in the yaml but that did not update the default port for default server which is still 80 as you pointed out.

[grk-pancham]

Hi Rulox,

I redeployed IC changing status port to 8081 (-nginx-status-port=8081). IC deployed correctly and no errors in the log. I do not see the ELB health check ping to ICC too. weird. but the default port for server is still 80. how do you i change the default port 80 ?

[Rulox]

@grk-kulkarni Yes you can change that port.

One solution is to change the hostPort in your manifest instead of 80 you can use 8080 there (if you are using -nginx-status-port=8081 to avoid conflict).

But the solution depends on how you have deployed your IC, would this work for you? If not, I need to know how you are deploying the IC (daemonset/deployment, what service type, etc)

[grk-pancham]

Hi Rulox,

I am deploying IC as daemonset.

I changed the hostport and containerport to 8080 in manifest file and IC still uses the default port 80. Not sure how to change that. I was also using status port to 8081 using -nginx-status-port=8081 to avoid conflict.

I removed the forward port of 80 to 8080 from firewall rules and I was able to make IC work successfully.

I do not think we can change the port forward from firewall rules so the question is how to change the default port 80 & 443 for IC ?

[Rulox]

You only need to change the hostPort: 80, you can't change the containerPort because the IC pod will always serve in 80 and 443, that's hardcoded in the NGINX Conf.

So your daemonset will look like:

spec:
      serviceAccountName: nginx-ingress
      containers:
      - image: nginx/nginx-ingress:edge
        imagePullPolicy: Always
        name: nginx-ingress
        ports:
        - name: http
          containerPort: 80
          hostPort: 8080

That means, the IC will use the port 8080 in your host (node) and will redirect to the 80 in the pod. Will this work for you? I think is what you want

[grk-pancham]

Hi Rulox,

I was able to successfully test IC and nifi service on port 8080. However I could not test IC on port 8443. I have updated the default secret with my SSL certificate. I do not get any errors in the log and the nifi ingress also looks good.

However I get 404 Not Found response when running IC on 8443. My nifi service is running on port 8080 which i think should not be a problem.

when i do curl on https://NODE_IP:8443/nifi, I get SSL certificate verification failed. when I add option -k to skip ssl verification, I get 404 Not Found.

Also the IC health check on port 8443 is working fine and IC nodes are healthy on the ALB target group. my SSL certificate is valid.

Does the backend service nifi also have to run on port 8443 to make this work. I do not think so.

this the generated nginx ingress config by IC `# configuration for default/nifi-ingress

upstream default-nifi-ingress-mydomain.com-nifi-8080 { zone default-nifi-ingress-mydomain.com-nifi-8080 256k; random two least_conn;

server 192.168.214.193:8080 max_fails=1 fail_timeout=10s max_conns=0;

}

server {

listen 80;

server_tokens on;

server_name mydomain.com;

location /nifi {

    proxy_http_version 1.1;

    proxy_connect_timeout 10s;
    proxy_read_timeout 10s;
    proxy_send_timeout 60s;
    client_max_body_size 2m;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering on;

    proxy_pass http://default-nifi-ingress-mydomain.com-nifi-8080;

}

}`

[Rulox]

What is your daemonset manifest? and your Ingress manifest?

[grk-pancham]

Hi Rulox,

Attached is the IC deployment manifest, nifi ingress deployment manifest, nginx config and nifi ingress config.

Let me know if you need any other details.

default-nifi-ingress.conf.txt nifi-ingress.yaml.txt nginx.conf.txt nginx-ingress.yaml.txt

[Rulox]

Your ingress need to specify TLS termination in order to use port 443. If no tls host is defined, NGINX exposes only port 80 for your Ingress (for obvious security reasons).

If you check the config in default-nifi-ingress.conf.txt server block starts with:

server {
    listen 80;
    server_tokens on;
    server_name mydomain.com;
#... etc

So for your ingress, only port 80 is working, not 443.

You need to include your secret in your Ingress Config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nifi-ingress
spec:
  tls:
  - hosts:
    - mydomain.com
    secretName: <YOUR_SECRET_NAME>
  rules:
  - host: mydomain.com
    http:
      paths:
      - path: /nifi
        backend:
          serviceName: nifi
          servicePort: 8080

This way, the Ingress Controller will also make NGINX to expose port 443, and because you have the hostPort 8443 to use the 443 of your IC pod, everything should work.

I suggest you to carefully check the Ingress documentation to understand how it works.

Hope it helps!

[grk-pancham]

Hi Rulox,

I know I am missing the TLS in ingress. But i am not clear on the below scenario.

If you look at the manifest I provided, my IC is running on port 8080 and 8443. I am able to access IC on port 8080 via external ALB successfully but not able to access IC on port 8443. I get response 404 not found. what is the reason for that. I thought the ingress comes later when the IC uses ingress rules to redirect/forward request to backend services but I am not clear as to why IC itself is not accessible on port 8443 ? Does the missing TLS in ingress rule cause IC to only exposes itself on port 8080 and not on port 8443 ?

I thought the way it works is that IC is listening to port 8080/8443 on the node for requests. It then uses the ingress rules to proxy the requests to backend service as defined by ingress rules.

But i will add TLS and also update you on the results.

[Rulox]

Hey @grk-kulkarni let me try to explain how the Ingress Controller works:

Scenario 1. You deploy the Ingress Controller by default (no ingress, no configmap, nothing else).

• All requests to 80 and 443 will return a 404, because there is no rule defined for any Ingress Resource declared. The idea of the Ingress Controller is to Load Balance Ingress resources in your cluster.

Scenario 2. You deploy the Ingress Controller with the health-status to true (no ingress, no configmap..):

• You can access your IC_IP:80/nginx-health (8080 in your case) or IC_IP:443/nginx-health (8443 for you)
• You still get 404 for any other route, including /, again, the same reason.. there are no Ingress Resources defined for this Ingress Controller.

Scenario 3. You deploy the Ingress Controller with healht-status to true, AND one Ingress Resource without TLS defined (this is your case right now):

• You can access your IC_IP:80/nginx-health (8080 in your case) or IC_IP:443/nginx-health (8443 for you)
• You can access your IC_IP:80/your_ingress (8080 in your case)
• You get a 404 if you access IC_IP:443/your_ingress (8443) because NGINX is not listening to that port for that specific ingress.

In summary, NGINX is always listening in 80 and 443, the difference is that NGINX will return a 404 if the defined rule for that Ingress is not present. You need to specify explicitly how your Ingress work. If you want to use port 443 (8443) with your ingresses, you must specify a TLS for that Ingress. Every Ingress is independent to the other and to the global config.

Let me know if it makes sense, I tried to explain it as easy as I could!

[grk-pancham]

Hi Rulox,

that makes sense. Thanks for the detailed explanation.

I updated the TLS on the nifi ingress and as expected IC and nifi service is now working successfully on port 8443. thanks for all your help.

i had a side question. Is it possible to use ingress to proxy backend services that are outside Kubernetes cluster.

[Rulox]

Yes, you can loadbalance to external services using ExternalName services. However, this feature requires NGINX Plus, the paid version of NGINX, because it uses the DNS resolve in realtime feature, which is only for Plus. Check this doc for more info.

Another solution could be redirect using snippets annotation, but this wouldn't be the same, as you are redirecting the request, not balancing it.

If that's all, feel free to close the issue, thanks!

[grk-pancham]

one last question on the ExternalName feature. By NGINX plus , you mean plus edition of NGINX ingress controller for Kubernetes Plus version and NOT the separate product NGINX plus that is available from NGINX.

[magicalyak]

@grk-kulkarni You would get an NGINX Plus license and use that for the NGINX Plus Ingress Controller. Might be easier to think of it as using "NGINX Plus" for "Kubernetes Ingress" (aka NGINX Plus Kubernetes Ingress Controller). There isn't a unique license for NGINX Plus KIC, it's just using NGINX Plus for the KIC.

[Rulox]

Closing this, feel free to re-open if issue persists.