[Feature] allocateLoadBalancerNodePorts field supports from AKS

euchungmsft commented 1 year ago

Is your feature request related to a problem? Please describe. Avoiding node port allocation for better performance by allocateLoadBalancerNodePorts = false which has been supported from 1.24

Check this out https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#service-v1-core

Describe the solution you'd like allocateLoadBalancerNodePorts field supports in service deployment

Describe alternatives you've considered Tested custom service controller settings but didn't work at all. Seems like there's nothing to do with it

Additional context

Competitors has been supporting it.
Apparently the situation looks the same with this issue here but there seems nothing to do with health probes

Repro

simple-node-service.yml

apiVersion: v1
kind: Service
metadata:
  name: simple-node-app
spec:
  allocateLoadBalancerNodePorts: false
  type: LoadBalancer
  ports:
  - port: 8000
  selector:
    app: simple-node-app

Command :

prompt> kubectl apply -f simple-node-service.yml

Error :

prompt> kubectl describe services simple-node-app

Name:                     simple-node-app
Namespace:                default
Labels:                   <none>
<ignore this>
Annotations:              service.beta.kubernetes.io/azure-load-balancer-internal: true
                          service.beta.kubernetes.io/port_443_health-probe_port: 8000
                          service.beta.kubernetes.io/port_443_health-probe_protocol: http
Selector:                 app=simple-node-app
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.0.216.124
IPs:                      10.0.216.124
Port:                     <unset>  8000/TCP
TargetPort:               8000/TCP
Endpoints:                10.244.1.3:8000
Session Affinity:         None
External Traffic Policy:  Cluster

Events:
  Type     Reason                  Age              From                Message
  ----     ------                  ----             ----                -------
  Normal   EnsuringLoadBalancer    4s (x2 over 9s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  3s (x2 over 9s)  service-controller  Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {\r
  "error": {\r
    "code": "PortValueIsOutOfRange",\r
    "message": "Resource a88de9d5db5104368a62f9c23a1342b9-TCP-8000 has invalid value of Port (0). The value must be between 1 and 65535.",\r
    "details": []\r
  }\r
}

On AKS portal

mikeoleary commented 10 months ago

@euchungmsft I am just coming across this now as I've hit the same issue as you.

However, I read here that if you set allocateLoadBalancerNodePorts: false you must also set External Traffic Policy: Local

When allocateLoadBalancerNodePorts is set to false, ensure the following conditions are met: Set externalTrafficPolicy to Local. And enable the FloatingIP feature by either not setting annotation service.beta.kubernetes.io/azure-disable-load-balancer-floating-ip, or setting its value to false

I have updated my spec and my Azure LB is now updated successfully.

mikeoleary commented 9 months ago

Update, in case you are following along, @euchungmsft

I had hit the same error as you, and I after I used both attributes in my YAML manifest, allocateLoadBalancerNodePorts: false and externalTrafficPolicy: Local, I was able to successfully deploy my Azure LB.

I am now doing this all again to verify and I am still seeing that my Azure LB backend pool contains the Node IP addresses, not the Pod IP addresses I am expecting to see. Like you refer to in your first few lines of this issue, I'm trying to have the load balancer send directly to pods, bypassing kube-proxy for improved performance. Reading further, I see a way to set loadBalancerBackendPoolConfigurationType to either NodeIPConfiguration, NodeIP, or PodIP, however it appears PodIP is "not supported yet".

As best I can tell then, Azure still does not support load balancing directly to pod IP addresses. Does anybody know if that is correct, and if/when updates are expected? I would love some advice or documentation from AKS PM.