ADC/ALB cant reach the pod endpoints , ClusterIP network

[iahmad-khan]

Use case:

An app running on Kubernetes needs to be exposed outside via ingress resources and external ADC

Steps:

• deploy ingress controller and CRDS on Kubernetes, ingress controller can talk to external ADC/ALB ( ADC running as appliance on VMware ) , done
• ingress controller can talk to ADC via REST, done
• deploy httpbin app and service, done
• create ingress resource for httpbin, done
• ADC is updated by ingress controller with a virtual service and real server backend, done
• ADC can't reach the backend pods/IPs, this needs fixing.

Possible solution:

the Ingress controller should register the Kubernetes node-port service in ADC instead of Pod IPs/endpoints. Pod/IPs are internal to Kubernetes and cant be reached from external ADC

[ghowett]

Currently the system will retrieve the POD/IP from the ingress controller. These IP's are not accessible externally. The system should find the associated POD IPS's Endpoint that is an external IP:port that is contactable externally. We would like to change this as a config setting PodEndPoints, ServiceIPs. Default is the new mode ServiceIPs

[iahmad-khan]

In short, currently the ingress controller is using PodEndpoints, but for external ADC, we need NodePort service based IPs and ports. NodePort service IPs and ports are accessible from ADC

[SmartCodeMaker]

@ghowett @iahmad-khan This is my actual controller service for edgenexus-ingress (of type of LoadBalancer):

# Source: edgenexus-ingress/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: edgenexus-ingress
  namespace: edgenexus-ingress
  labels:
    app.kubernetes.io/name: edgenexus-ingress
    helm.sh/chart: edgenexus-ingress-0.10.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: my-release
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  loadBalancerIP: 192.168.2.132
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    app:  edgenexus-ingress

Do you think we need to create NodePort instead of LoadBalancer type?

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app.kubernetes.io/name: edgenexus-ingress
    helm.sh/chart: edgenexus-ingress-0.10.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: my-release
    app.kubernetes.io/component: controller
  name: edgenexus-ingress
  namespace: edgenexus-ingress
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      appProtocol: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      appProtocol: https
  selector:
    app:  edgenexus-ingress

If so, then what should ingress controller do to ADC?

[iahmad-khan]

@SmartCodeMaker the issue is not related to the service type of controller.

The issue is the way the ingress controller is discovering the pods and registering them with ADC. for example, I deploy an app, and that app is registered with ADC by the ingress controller, but the ADC can see only the pod internal IPs or endpoints, ADC cant see the nodeport service for the app.

When ADC sees the pod endpoints, it can't reach them.

If we make ADC see the nodeport ips instead, ADC will be able to access the backend app running on kubernetes.

[SmartCodeMaker]

@iahmad-khan Could you take a look at this video https://easyupload.io/shknyg and the video attached? There's an internal processing shown. And tell me what addresses you'd like to register with the ADC? Or what discovering (and at which step) should be performed by the ingress controller.

https://github.com/edgeNEXUS/kubernetes-ingress/assets/81356751/c887b23e-eea0-420c-9730-86594ce36174

[iahmad-khan]

@SmartCodeMaker yes, the list of addresses for REAL servers, the 10.x.x.x network ips, those are internal pod ips that cant be accessed from external ADC. we need nodePort Ips and ports under real servers section for each VS