LoadBalancer Service with multiple ports creates multiple Virtual Services

[an0nz]

Describe the bug

When you create a LoadBalancer service in k8s with multiple ports such as below this generates multiple virtual services in Cloud Director instead of a single virtual service with multiple ports.

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-ingress-controller
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http
      nodePort: 32697
    - name: https
      protocol: TCP
      port: 443
      targetPort: https
      nodePort: 30032
  selector:
    app: nginx-ingress
  type: LoadBalancer

Results in the following virtual services being added to vCD

ingress-vs-ingress-nginx-ingress-controller-____-http     Virtual IP: x.x.x.1     Ports: 80 (L4)
ingress-vs-ingress-nginx-ingress-controller-____-https    Virtual IP: x.x.x.2     Ports: 443 (L4)

From the vCD UI you cannot add multiple ports either other than using a range in the TCP Proxy field such as 80-443.

However multiple ports do seem to work as if you edit the Virtual Service in NSX-ALB adding the additional port this is displayed correctly in vCD where another TCP Proxy appears with an x to delete it and it can be modified and saved, the UI just doesn't look to have a button for adding additional TCP Proxy ports at this time but the API seems to function correctly.

The API call vCD does to retrieve and display multiple ports has the following structure for servicePorts

https://VCD_FQDN/cloudapi/1.0.0/edgeGateways/urn:vcloud:gateway:GUID/loadBalancer/virtualServiceSummaries?page=1&pageSize=15&sortAsc=name&links=true
"servicePorts": [{
                    "tcpUdpProfile": {
                        "name": null,
                        "type": "-",
                        "systemDefined": null
                    },
                    "portStart": 80,
                    "portEnd": 80,
                    "sslEnabled": false
                }, {
                    "tcpUdpProfile": {
                        "name": null,
                        "type": "-",
                        "systemDefined": null
                    },
                    "portStart": 443,
                    "portEnd": 443,
                    "sslEnabled": false
                }
            ],

When I try to edit the port in vCD though the API returns an error stating Edge Gateway k8s-edge can have multiple service ports only with additional licensing. Please contact your service provider.

None of this makes exact sense as AVI seems to support multiple ports under the license however vCD rejects you when you edit it so maybe this is not possible.

Reproduction steps

Cloud Director: 10.3.2
NSX-T: 3.1.3.1
AVI (NSX-ALB): 21.1.2 with Basic License
CCM: Built from main branch

1. Create a load balancer service in vCD with multiple port definitions
2. Observe that multiple virtual services are created in NSX-ALB when there should only be one

Expected behavior

A single virtual service with multiple L4 ports is created, instead of multiple services.

Additional context

It seems like a fairly fundamental feature for a single IP to be able to listen on multiple ports, especially when talking about HTTP and HTTPS as these cannot be on different IP addresses else a website may not work correctly.

[an0nz]

Correction to the previous, this does work from a public standpoint as DNAT rules for the same public IP are setup to point to the individual L4 services. It is wasteful from a Service Engine perspective as every HTTP/HTTPS L4 load balancer requires 2 Virtual Services each consuming an internal IP and on an AVI basic license each Service Engine can only run 10 virtual services so it ends up using more compute power than necessary.

[arunmk]

@an0nz your analysis is correct. However this was a deliberate design aspect that one service should not overwhelm the other service and there must be some amount of isolation between the two.

Do you think there should be a flag to gate this, so that the users could choose to have one Virtual service for all of the services in one ingress?

[arunmk]

@an0nz currently VCD does not support single virtual service with multiple ports and load balancers on a Basic Avi license. This will get added as soon as the feature is added into VCD. I will convert this into an enhancement request.

[arunmk]

@an0nz, a commit was recently added by @ltimothy7 which would allow one to use a single IP address and multiple virtual services. So there would be no DNAT rule and the same VIP would be used across multiple virtual services.

This needs VCD 10.4.0 + which is to be released shortly.

Would that satisfy your requirement?

[an0nz]

@arunmk, thanks for the update this will work nicely to avoid DNAT, it sounds like this still uses multiple Virtual Services which will result in twice the number of VM’s needed to run NSX-ALB service engines as most load balancer use cases we see are HTTP/HTTPS if you could put both on the same VS that would be ideal. I look forward to testing this once everything is released.

[ltimothy7]

Thank you @an0nz; using the same VS for HTTP/HTTPS is not currently planned, but we have it on our list of features to look into. We will keep this thread updated with any plans