Supports multi-port services using a single IP address

[6547709]

Issue Description:

I am using an external LoxiLB setup(kube-loxilb in kubernetes), and when creating a Service for Ingress-Nginx, regardless of the “setUniqueIP” configuration, LoxiLB assigns two IP addresses to the Service because it has two ports (80 and 443). Since Ingress access involves DNS resolution of domain names, it’s impractical to have two IP addresses, one for port 80 and another for port 443.

Temporary Solution:

I can temporarily solve this issue by specifying an IP address outside of IPAM using "loxilb.io/staticIP", which allows multi-port services to use a single IP address.

Suggestion:

Could you adjust the code so that for services with multiple ports, LoxiLB assigns only one IP address?

Desired results: $ docker exec -it loxilb loxicmd get lb	EXT IP	PORT	PROTO	NAME	MARK	SEL	MODE	# OF ENDPOINTS	MONITOR
10.60.5.1	80	tcp	ingress-nginx_ingress-nginx-controller	0	hash	onearm	4	On
10.60.5.1	443	tcp	ingress-nginx_ingress-nginx-controller	0	hash	onearm	4	On

[TrekkieCoder]

Hi @6547709 Can you please provide kube-loxilb.yaml contents (the arguments) and also the service yaml ? It would help check the issue more. AFAIK, this should already be supported.

[6547709]

@TrekkieCoder Thank you for your reply, here are the relevant configurations. Kube-LoxiLB Args

      containers:
      - name: kube-loxilb
        image: ghcr.io/loxilb-io/kube-loxilb:latest
        imagePullPolicy: Always
        command:
        - /bin/kube-loxilb
        args:
        - --loxiURL=http://10.40.45.8:11111,http://10.40.45.9:11111
        - --externalCIDR=10.60.4.0/24
        - --monitor
        - --setLBMode=2
        - --setUniqueIP=true

Ingress-Nginx Service (Use Helm Install)

apiVersion: v1
kind: Service
metadata:
  annotations:
    loxilb.io/epselect: hash
    loxilb.io/lbmode: onearm
    loxilb.io/liveness: "yes"
    loxilb.io/proberetries: "3"
    loxilb.io/probetimeout: "10"
    loxilb.io/staticIP: 10.60.5.1
    loxilb.io/usepodnetwork: "yes"
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-nginx
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.2
    helm.sh/chart: ingress-nginx-4.11.2
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  loadBalancerClass: loxilb.io/loxilb
  ports:
  - appProtocol: http
    name: http
    nodePort: 30988
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: 30604
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer

External LoxiLB Docker(Use GARP for VIP)

docker run -u root --cap-add SYS_ADMIN --network=host \
  --restart always --privileged -dit \
  -v /dev/log:/dev/log \
  -v /etc/gobgp:/etc/gobgp \
  --name loxilb \
  ghcr.io/loxilb-io/loxilb:latest \
  --cluster=10.40.45.9 \
  --self=0 \
  --host=0.0.0.0 \
  --ka=10.40.45.9:10.40.45.8  \
  -b \
  -p

[TrekkieCoder]

I think if you remove - --setUniqueIP=true, it should work as expected. setUniqueIP tries to allocate a newIP to every multi-port sub-service as well.

[6547709]

I have tested it before and just tested it again, remove - --setUniqueIP=true， Still two IP addresses.

$ kubectl -n ingress-nginx get svc
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP                   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.107.220.68   llb-10.60.4.1,llb-10.60.4.3   80:31034/TCP,443:30955/TCP   6s

[TrekkieCoder]

Ok got it. Although my setup is not 100% same as yours but does not reproduce in mine :

kube-system   loxilb-ingress-manager   LoadBalancer   10.43.30.135   llb-192.168.80.1   80:30988/TCP,443:30604/TCP   3m39s

Can you please paste the kube-loxilb logs ? It will help understand the cause.

kubectl logs -f -n kube-system kube-loxilb-xxxxxx-xxxx

Update : I also tried with helm chart from ingress-nginx and just changed loadBalancerClass: "loxilb.io/loxilb" in values.yaml . Still could not reproduce. Will wait for kube-loxilb logs to debug further !

[6547709]

I conducted tests based on your configuration and obtained the same results as you. Note: I completely removed the deployment of kube-loxilb and then redeployed kube-loxilb and Ingress-Nginx, and it worked properly.

Now, the question should be 'How to support multi port services using a single IP address with' setUniqueIP=true '.

I hope each service has a separate IP address instead of multiple services sharing one IP address;

[TrekkieCoder]

I conducted tests based on your configuration and obtained the same results as you. Note: I completely removed the deployment of kube-loxilb and then redeployed kube-loxilb and Ingress-Nginx, and it worked properly.

Ok great.

Now, the question should be 'How to support multi port services using a single IP address with' setUniqueIP=true '.

I hope each service has a separate IP address instead of multiple services sharing one IP address;

Yes it makes sense for setUniqueIP to work as expected for multi-port service. Will work on tweaking the logic. Should not be a big effort.

[6547709]

Thank you for confirming this problem together. I look forward to the adjustment of the code.

[TrekkieCoder]

Now, kube-loxilb will make sure to allocate IP properly even if there are multi-port services. setUniqueIP will allocate unique IP on a per-service basis but not on a sub-rule basis. Please redeploy kube-loxilb/services and check.