failed to generate external IP. IP Pool is full

Image: ghcr.io/loxilb-io/kube-loxilb:latest Args:

--loxiURL=http://172.24.25.207:11111,http://172.24… --externalCIDR=172.24.25.0/24 --monitor --setLBMode=2

Deploy rabbitmq svc with type LoadBalancer:

2023-09-10T05:45:17.695450181+07:00 I0909 22:45:17.692694       1 ippool.go:72] Allocate ServiceIP 172.24.25.1:5672 (tcp)
2023-09-10T05:45:17.695479596+07:00 I0909 22:45:17.692724       1 ippool.go:72] Allocate ServiceIP 172.24.25.1:15672 (tcp)
2023-09-10T05:45:17.695486129+07:00 I0909 22:45:17.692730       1 ippool.go:72] Allocate ServiceIP 172.24.25.1:15692 (tcp)
2023-09-10T05:45:17.695491369+07:00 I0909 22:45:17.692739       1 loadbalancer.go:564] Endpoint IP Pairs [172.24.25.201 172.24.25.202 172.24.25.203]
2023-09-10T05:45:17.695500656+07:00 I0909 22:45:17.692746       1 loadbalancer.go:565] Secondary IP Pairs []
2023-09-10T05:45:17.695506808+07:00 I0909 22:45:17.694623       1 loadbalancer.go:656] load-balancer ([{{172.24.25.1 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.24.25.201 30212 4 } {172.24.25.202 30212 3 } {172.24.25.203 30212 3 }]}]) added
2023-09-10T05:45:17.696060636+07:00 I0909 22:45:17.695951       1 loadbalancer.go:656] load-balancer ([{{172.24.25.1 15672 tcp 0 2 false true 1800 0 true  0  } [] [{172.24.25.201 32523 4 } {172.24.25.202 32523 3 } {172.24.25.203 32523 3 }]}]) added
2023-09-10T05:45:17.697319748+07:00 I0909 22:45:17.697172       1 loadbalancer.go:656] load-balancer ([{{172.24.25.1 15692 tcp 0 2 false true 1800 0 true  0  } [] [{172.24.25.201 31979 4 } {172.24.25.202 31979 3 } {172.24.25.203 31979 3 }]}]) added

Deploy second rabbitmq svc, with only deference is svc name:

2023-09-10T05:47:20.271265524+07:00 E0909 22:47:20.271165       1 ippool.go:68] ip/ident exists
2023-09-10T05:47:20.271353628+07:00 E0909 22:47:20.271288       1 loadbalancer.go:913] failed to generate external IP. IP Pool is full
2023-09-10T05:47:20.271757526+07:00 E0909 22:47:20.271701       1 loadbalancer.go:258] Error syncing Node {default production-rabbitmqcluster-lb2}, requeuing. Error: failed to generate external IP. IP Pool is full

Why there is error failed to generate external IP. IP Pool is full ?? externalCIDR is 172.24.25.0/24, there are 254 host available with cidr /24, why loxilb not used ip172.24.25.2 as LoadBalancer ??

Deploy different svc with different port:

2023-09-10T05:56:57.143892381+07:00 I0909 22:56:57.143607       1 ippool.go:72] Allocate ServiceIP 172.24.25.1:80 (tcp)
2023-09-10T05:56:57.143949258+07:00 I0909 22:56:57.143634       1 loadbalancer.go:564] Endpoint IP Pairs [172.24.25.201 172.24.25.202 172.24.25.203]
2023-09-10T05:56:57.143955199+07:00 I0909 22:56:57.143641       1 loadbalancer.go:565] Secondary IP Pairs []
2023-09-10T05:56:57.144990395+07:00 I0909 22:56:57.144858       1 loadbalancer.go:656] load-balancer ([{{172.24.25.1 80 tcp 0 2 false true 1800 0 true  0  } [] [{172.24.25.201 31111 4 } {172.24.25.202 31111 3 } {172.24.25.203 31111 3 }]}]) added

It seems that loxilb only used first available host from externalCIDR 172.24.25.0/24 which is 172.24.25.1 and loxilb can not used next available host ip. This is like port overloading, 1 ip for multiple service (port)

With metallb layer2, i can create multiple svc with same/different port with different name and metallb layer2 will select next available ip from ip pool,

Need help and advice, thank you.

Request to do the following :

Kindly update to latest kube-loxilb image(Delete and redeploy after changing yaml because some fixes were made in this regard)
Use - --setUniqueIP=true flag in kube-loxilb yaml. This means it will try to allocate and use unique IP for each service.

Thank you, now kube-loxilb can utilized unique ip,

2023-09-14T07:09:21.075920337+07:00 I0914 00:09:21.074676       1 loadbalancer.go:662] load-balancer ([{ {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}]) added
2023-09-14T07:09:21.084927336+07:00 I0914 00:09:21.083534       1 loadbalancer.go:662] load-balancer ([{ {{172.22.4.1 15672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32637 2 } {172.22.4.105 32637 2 } {172.22.4.106 32637 2 } {172.22.4.107 32637 2 } {172.22.4.108 32637 1 } {172.22.4.109 32637 1 }]}}]) added
2023-09-14T07:09:21.091655419+07:00 I0914 00:09:21.091433       1 loadbalancer.go:662] load-balancer ([{ {{172.22.4.2 15692 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 30707 2 } {172.22.4.105 30707 2 } {172.22.4.106 30707 2 } {172.22.4.107 30707 2 } {172.22.4.108 30707 1 } {172.22.4.109 30707 1 }]}}]) added
2023-09-14T07:09:21.304246222+07:00 E0914 00:09:21.304064       1 loadbalancer.go:675] failed to update service production-rabbitmqcluster-lb.status. err: Operation cannot be fulfilled on services "production-rabbitmqcluster-lb": the object has been modified; please apply your changes to the latest version and try again
2023-09-14T07:10:06.169484616+07:00 I0914 00:10:06.166951       1 ippool.go:81] Allocate ServiceIP 172.22.4.3:5672 (tcp)
2023-09-14T07:10:06.169499974+07:00 I0914 00:10:06.167273       1 ippool.go:81] Allocate ServiceIP 172.22.4.4:15672 (tcp)
2023-09-14T07:10:06.169504453+07:00 I0914 00:10:06.167285       1 ippool.go:81] Allocate ServiceIP 172.22.4.5:15692 (tcp)
2023-09-14T07:10:06.169509202+07:00 I0914 00:10:06.167294       1 loadbalancer.go:570] Endpoint IP Pairs [172.22.4.108 172.22.4.109 172.22.4.104 172.22.4.105 172.22.4.106 172.22.4.107]
2023-09-14T07:10:06.169513971+07:00 I0914 00:10:06.167404       1 loadbalancer.go:571] Secondary IP Pairs []
2023-09-14T07:10:06.186924307+07:00 I0914 00:10:06.172003       1 loadbalancer.go:662] load-balancer ([{ {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}]) added
2023-09-14T07:10:06.208596699+07:00 I0914 00:10:06.203373       1 loadbalancer.go:662] load-balancer ([{ {{172.22.4.4 15672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31287 2 } {172.22.4.109 31287 2 } {172.22.4.104 31287 2 } {172.22.4.105 31287 2 } {172.22.4.106 31287 1 } {172.22.4.107 31287 1 }]}}]) added
2023-09-14T07:10:06.212880952+07:00 I0914 00:10:06.211896       1 loadbalancer.go:662] load-balancer ([{ {{172.22.4.5 15692 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 30786 2 } {172.22.4.109 30786 2 } {172.22.4.104 30786 2 } {172.22.4.105 30786 2 } {172.22.4.106 30786 1 } {172.22.4.107 30786 1 }]}}]) added

But there are new issues, I tested with - --externalCIDR=172.22.4.0/26, for rabbitmq there are 3 port 5672, 15672 and 15692.

For first svc1 deployed used ip 172.22.4.0, 172.22.4.1 and 172.22.4.2 For second svc2 deployed used ip 172.22.4.3, 172.22.4.4 and 172.22.4.5 Can't access rabbitmq ui port 15672,

I deleted svc1 and svc2, redeploy again svc1 and i got ip 172.22.4.6, 172.22.4.7 and 172.22.4.8 kube-loxilb not used empty ip addr, it just keep on continue used sequence ip host from CIDR and not considered ip from CIDR which has been relesed, And i still can't access rabbitmq 15672

kube-loxilb log:

2023-09-14T07:18:47.076544816+07:00 I0914 00:18:47.075475       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:18:47.076916191+07:00 I0914 00:18:47.075552       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:18:47.076937381+07:00 I0914 00:18:47.075643       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:18:47.077811826+07:00 E0914 00:18:47.076791       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb2}, requeuing. Error: failed to delete loxiLB LoadBalancer
2023-09-14T07:18:57.129005679+07:00 I0914 00:18:57.128794       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:18:57.129485564+07:00 I0914 00:18:57.129179       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:18:57.129565153+07:00 I0914 00:18:57.128937       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:18:57.130894399+07:00 E0914 00:18:57.130814       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb}, requeuing. Error: failed to delete loxiLB LoadBalancer
2023-09-14T07:19:51.079900796+07:00 I0914 00:19:51.077939       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:19:51.080942091+07:00 I0914 00:19:51.078069       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:19:51.081008305+07:00 I0914 00:19:51.078097       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:19:51.081133029+07:00 E0914 00:19:51.079676       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb2}, requeuing. Error: failed to delete loxiLB LoadBalancer
2023-09-14T07:20:01.132811782+07:00 I0914 00:20:01.132262       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:20:01.132880010+07:00 I0914 00:20:01.132389       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:20:01.132887334+07:00 I0914 00:20:01.132522       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:20:01.134794753+07:00 E0914 00:20:01.134625       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb}, requeuing. Error: failed to delete loxiLB LoadBalancer
2023-09-14T07:21:51.080890337+07:00 I0914 00:21:51.080333       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:21:51.081675909+07:00 I0914 00:21:51.080476       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:21:51.081706987+07:00 I0914 00:21:51.080357       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:21:51.083321482+07:00 E0914 00:21:51.083162       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb2}, requeuing. Error: failed to delete loxiLB LoadBalancer
2023-09-14T07:22:01.135849868+07:00 I0914 00:22:01.135186       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:22:01.136811836+07:00 I0914 00:22:01.135484       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:22:01.136853454+07:00 I0914 00:22:01.135538       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:22:01.137165959+07:00 E0914 00:22:01.137044       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb}, requeuing. Error: failed to delete loxiLB LoadBalancer
2023-09-14T07:23:51.083823386+07:00 I0914 00:23:51.083574       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:23:51.084390388+07:00 I0914 00:23:51.083763       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:23:51.084727049+07:00 I0914 00:23:51.083761       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:23:51.086937718+07:00 E0914 00:23:51.086552       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb2}, requeuing. Error: failed to delete loxiLB LoadBalancer
2023-09-14T07:24:01.140664768+07:00 I0914 00:24:01.138282       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:24:01.140756330+07:00 I0914 00:24:01.138584       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:24:01.140767280+07:00 I0914 00:24:01.138837       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:24:01.140774554+07:00 E0914 00:24:01.140244       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb}, requeuing. Error: failed to delete loxiLB LoadBalancer
2023-09-14T07:25:51.087836319+07:00 I0914 00:25:51.087076       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:25:51.088054273+07:00 I0914 00:25:51.087331       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:25:51.088073469+07:00 I0914 00:25:51.087506       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.3 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.108 31356 2 } {172.22.4.109 31356 2 } {172.22.4.104 31356 2 } {172.22.4.105 31356 2 } {172.22.4.106 31356 1 } {172.22.4.107 31356 1 }]}}
2023-09-14T07:25:51.089381010+07:00 E0914 00:25:51.089097       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb2}, requeuing. Error: failed to delete loxiLB LoadBalancer
2023-09-14T07:26:01.143132111+07:00 I0914 00:26:01.141400       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:26:01.143257500+07:00 I0914 00:26:01.141438       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:26:01.143265925+07:00 I0914 00:26:01.141401       1 loadbalancer.go:705] called loxilb API: delete lb rule { {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.104 32408 2 } {172.22.4.105 32408 2 } {172.22.4.106 32408 2 } {172.22.4.107 32408 2 } {172.22.4.108 32408 1 } {172.22.4.109 32408 1 }]}}
2023-09-14T07:26:01.145725414+07:00 E0914 00:26:01.144953       1 loadbalancer.go:264] Error syncing Node {default production-rabbitmqcluster-lb}, requeuing. Error: failed to delete loxiLB LoadBalancer

@4ss3g4f

Thanks for the update. Please let us know the following info:

Can you please let us know the contents of yaml files for service creation ?
Also let us know the output of loxicmd get lb -o wide from loxilb ?

Also, IPAM allocation is incremental in loxilb. It will reuse returned addressed when all incremental IPAM addresses are exhausted. So, that part is fine.

svc-loxilb1.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
   # If there is a need to do liveness check from loxilb
   loxilb.io/liveness: "yes"
   # Specify LB mode - one of default, onearm or fullnat 
   loxilb.io/lbmode: "default"
   # Specify loxilb IPAM mode - one of ipv4, ipv6 or ipv6to4 
   loxilb.io/ipam: "ipv4"
   # Specify number of secondary networks for multi-homing
   # Only valid for SCTP currently
   # loxilb.io/num-secondary-networks: "2
  labels:
    app.kubernetes.io/name: production-rabbitmqcluster
  name: production-rabbitmqcluster-lb1
  namespace: default
spec:
  ports:
  - appProtocol: amqp
    name: amqp
    port: 5672
    protocol: TCP
    targetPort: 5672
  - appProtocol: http
    name: management
    port: 15672
    protocol: TCP
    targetPort: 15672
  - appProtocol: prometheus.io/metrics
    name: prometheus
    port: 15692
    protocol: TCP
    targetPort: 15692
  selector:
    app.kubernetes.io/name: production-rabbitmqcluster
  sessionAffinity: None
  type: LoadBalancer
  loadBalancerClass: loxilb.io/loxilb

svc-loxilb2.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
   # If there is a need to do liveness check from loxilb
   loxilb.io/liveness: "yes"
   # Specify LB mode - one of default, onearm or fullnat 
   loxilb.io/lbmode: "fullnat"
   # Specify loxilb IPAM mode - one of ipv4, ipv6 or ipv6to4 
   loxilb.io/ipam: "ipv4"
   # Specify number of secondary networks for multi-homing
   # Only valid for SCTP currently
   # loxilb.io/num-secondary-networks: "2
  labels:
    app.kubernetes.io/name: production-rabbitmqcluster
  name: production-rabbitmqcluster-lb2
  namespace: default
spec:
  ports:
  - appProtocol: amqp
    name: amqp
    port: 5672
    protocol: TCP
    targetPort: 5672
  - appProtocol: http
    name: management
    port: 15672
    protocol: TCP
    targetPort: 15672
  - appProtocol: prometheus.io/metrics
    name: prometheus
    port: 15692
    protocol: TCP
    targetPort: 15692
  selector:
    app.kubernetes.io/name: production-rabbitmqcluster
  sessionAffinity: None
  type: LoadBalancer
  loadBalancerClass: loxilb.io/loxilb

loxicmd get lb -o wide from loxilb 172.22.4.110

root@dr-worker-007:/# loxicmd get lb -o wide
| EXTERNAL IP | SECONDARY IPS | PORT  | PROTOCOL | BLOCK | SELECT |  MODE   | ENDPOINT IP  | TARGET PORT | WEIGHT | STATE  |
|-------------|---------------|-------|----------|-------|--------|---------|--------------|-------------|--------|--------|
| 172.22.4.6  |               |  5672 | tcp      |     0 | rr     | default | 172.22.4.104 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31258 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31258 |      1 | active |
| 172.22.4.7  |               | 15672 | tcp      |     0 | rr     | default | 172.22.4.104 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31493 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31493 |      1 | active |
| 172.22.4.8  |               | 15692 | tcp      |     0 | rr     | default | 172.22.4.104 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       30955 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       30955 |      1 | active |

loxicmd get lb -o wide from loxilb 172.22.4.111

| EXTERNAL IP | SECONDARY IPS | PORT  | PROTOCOL | BLOCK | SELECT |  MODE   | ENDPOINT IP  | TARGET PORT | WEIGHT | STATE  |
|-------------|---------------|-------|----------|-------|--------|---------|--------------|-------------|--------|--------|
| 172.22.4.6  |               |  5672 | tcp      |     0 | rr     | default | 172.22.4.104 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31258 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31258 |      1 | active |
| 172.22.4.7  |               | 15672 | tcp      |     0 | rr     | default | 172.22.4.104 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31493 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31493 |      1 | active |
| 172.22.4.8  |               | 15692 | tcp      |     0 | rr     | default | 172.22.4.104 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       30955 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       30955 |      1 | active |

error log from loxilb 172.22.4.111

Sep 14 04:53:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:53:52 [API] lbRules : {{172.22.4.3 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:53:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:53:52 [API] Error occur : no-rule error
Sep 14 04:53:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:53:52 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.3/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:53:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:53:52 [API] lbRules : {{172.22.4.3 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:53:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:53:52 [API] Error occur : no-rule error
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Error occur : no-rule error
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Error occur : no-rule error
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Error occur : no-rule error
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Error occur : no-rule error
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:54:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:54:02 [API] Error occur : no-rule error
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.3/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] lbRules : {{172.22.4.3 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Error occur : no-rule error
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.3/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] lbRules : {{172.22.4.3 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Error occur : no-rule error
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.3/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] lbRules : {{172.22.4.3 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Error occur : no-rule error
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.3/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] lbRules : {{172.22.4.3 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Error occur : no-rule error
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.3/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] lbRules : {{172.22.4.3 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:55:52 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:55:52 [API] Error occur : no-rule error
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Error occur : no-rule error
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Error occur : no-rule error
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Error occur : no-rule error
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Error occur : no-rule error
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Load balancer DELETE API called. url : /netlox/v1/config/loadbalancer/externalipaddress/172.22.4.0/port/5672/protocol/tcp?bgp=false&block=0
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false false 0 0 false  0  } [] []}
Sep 14 04:56:02 dr-worker-008.brids.co.id docker[1763]: 2023-09-13 21:56:02 [API] Error occur : no-rule error

loxicmd get lb -o wide from loxilb 172.22.4.112

| EXTERNAL IP | SECONDARY IPS | PORT  | PROTOCOL | BLOCK | SELECT |  MODE   | ENDPOINT IP  | TARGET PORT | WEIGHT | STATE  |
|-------------|---------------|-------|----------|-------|--------|---------|--------------|-------------|--------|--------|
| 172.22.4.6  |               |  5672 | tcp      |     0 | rr     | default | 172.22.4.104 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31258 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31258 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31258 |      1 | active |
| 172.22.4.7  |               | 15672 | tcp      |     0 | rr     | default | 172.22.4.104 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31493 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31493 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31493 |      1 | active |
| 172.22.4.8  |               | 15692 | tcp      |     0 | rr     | default | 172.22.4.104 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       30955 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       30955 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       30955 |      1 | active |

I used 2 loadbalancer loxilb and metallb layer2, from above screenshot we can see loxilb create 3 ip loadbalancers for 3 ports, while metallb only create 1 ip loadbalancer for 3 ports, this is important. from user stand point i don't need to check 3 ip for which ip is serving rabbitmq 15672, i only need to check 1 ip not 3, If the application expose multiple ports, let say 10 ports. There will be 10 ip, and need to check which ip for which port ???

And i can curl rabbitmq port 15672 from metallb layer2 ip loadbalancer,

core@dr-manager-01 ~/rabbitmq-operator $ curl 172.22.4.116:15672
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>RabbitMQ Management</title>
    <script src="js/ejs-1.0.min.js" type="text/javascript"></script>
    <script src="js/jquery-3.5.1.min.js"></script>
    <script src="js/jquery.flot-0.8.1.min.js" type="text/javascript"></script>
    <script src="js/jquery.flot-0.8.1.time.min.js" type="text/javascript"></script>
    <script src="js/sammy-0.7.6.min.js" type="text/javascript"></script>
    <script src="js/json2-2016.10.28.js" type="text/javascript"></script>
    <script src="js/base64.js" type="text/javascript"></script>
    <script src="js/global.js" type="text/javascript"></script>
    <script src="js/main.js" type="text/javascript"></script>
    <script src="js/prefs.js" type="text/javascript"></script>
    <script src="js/formatters.js" type="text/javascript"></script>
    <script src="js/charts.js" type="text/javascript"></script>
    <script src="js/oidc-oauth/helper.js"></script>
    <script src="js/oidc-oauth/oidc-client-ts.js" type="text/javascript"></script>
    <script src="js/oidc-oauth/bootstrap.js"></script>

    <link href="css/main.css" rel="stylesheet" type="text/css"/>
    <link href="favicon.ico" rel="shortcut icon" type="image/x-icon"/>

    <script type="application/javascript">
      var oauth = oauth_initialize_if_required();

      if (oauth.enabled) {
        if (!oauth.sp_initiated) {
            oauth.logged_in = has_auth_credentials();
            oauth.access_token = get_auth_credentials(); // DEPRECATED
        } else {
          oauth_is_logged_in().then( status => {
            if (status.loggedIn && !has_auth_credentials()) {
              oauth.logged_in = false;
              oauth_initiateLogout();
            } else {
              if (!status.loggedIn) {
                replace_content('outer', format('login_oauth', {}));
                clear_auth();
              } else {
                oauth.logged_in = true;
                oauth.access_token = status.user.access_token;  // DEPRECATED
                oauth.expiryDate = new Date(status.user.expires_at * 1000);  // it is epoch in seconds
                let current = new Date();
                _management_logger.debug('token expires in ', (oauth.expiryDate-current)/1000,
                  'secs at : ', oauth.expiryDate );
                oauth.user_name = status.user.profile['user_name'];
                if (!oauth.user_name || oauth.user_name == '') {
                  oauth.user_name = status.user.profile['sub'];
                }
                oauth.scopes = status.user.scope;
              }
            }
          });
        }
      }

    </script>

<!--[if lte IE 8]>
    <script src="js/excanvas.min.js" type="text/javascript"></script>
    <link href="css/evil.css" rel="stylesheet" type="text/css"/>
<![endif]-->
  </head>

  <body>
    <div id="outer"></div>
    <div id="debug"></div>
    <div id="scratch"></div>
  </body>
</html>
core@dr-manager-01 ~/rabbitmq-operator $

Got it. We will update soon !!

I'll be waiting, thank you for lightning speed response.

With additional fixes, we were to validate things working at our end. Some observations:

Unable to reproduce delete failed scenario
If multiple app-ports are specified per service, only a singleIP will be allocated as suggested
We usually use externalTrafficPolicy: Local in the LB spec to avoid unnecessary hops from loxilb to application pod. Not sure if that is causing some issue in your setup. Kindly double check.

Following is the yaml spec we tested with -

apiVersion: v1
kind: Service
metadata:
  name: nginx-service2
  annotations:
    loxilb.io/lbmode: "fullnat"
  labels:
    app: loxilb2
spec:
  externalTrafficPolicy: Local
  selector:
    app: loxilb2
  ports:
    - port: 10001
      targetPort: 80
      name: what
    - port: 20001
      targetPort: 80
      name: the
    - port: 30001
      targetPort: 80
      name: hell
  type: LoadBalancer
  loadBalancerClass: "loxilb.io/loxilb"
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx2
  labels:
    app: loxilb2
spec:
  containers:
  - name: nginx
    image: nginx:stable
    ports:
      - containerPort: 80
        name: http-web-svc

Services created -

$ sudo kubectl get svc
NAME             TYPE           CLUSTER-IP      EXTERNAL-IP         PORT(S)                                           AGE
kubernetes       ClusterIP      10.43.0.1       <none>              443/TCP                                           174m
nginx-service2   LoadBalancer   10.43.124.0     llb-123.123.123.5   10001:30135/TCP,20001:32544/TCP,30001:31523/TCP   3m53s
nginx-lb1        LoadBalancer   10.43.118.193   llb-123.123.123.6   55002:32562/TCP                                   2s

Further traffic to each IP:port combination was verified from external client. Kindly update and double confirm !!

excellent update, loxilb team is flash merged solution. I'll update at 8pm Jakarta Indonesia time, and double check also duplicate your svc config.

curl rabbitmq port 15672 still failed,

kube-loxilb log:

2023-09-15T04:57:09.026071091+07:00 I0914 21:57:09.025311       1 client.go:64] NewLoxiClient Created: http://172.22.4.110:11111
2023-09-15T04:57:09.026077183+07:00 I0914 21:57:09.025325       1 client.go:64] NewLoxiClient Created: http://172.22.4.111:11111
2023-09-15T04:57:09.026083264+07:00 I0914 21:57:09.025330       1 client.go:64] NewLoxiClient Created: http://172.22.4.112:11111
2023-09-15T04:57:09.026088975+07:00 I0914 21:57:09.025406       1 loadbalancer.go:227] Starting LoxilbLoadBalancerManager
2023-09-15T04:57:09.026094956+07:00 I0914 21:57:09.025422       1 shared_informer.go:273] Waiting for caches to sync for LoxilbLoadBalancerManager
2023-09-15T04:57:09.028219788+07:00 I0914 21:57:09.028093       1 client.go:80] LoxiHealthCheckChan: loxilb(http://172.22.4.112:11111) is alive
2023-09-15T04:57:09.028502357+07:00 I0914 21:57:09.028397       1 client.go:80] LoxiHealthCheckChan: loxilb(http://172.22.4.110:11111) is alive
2023-09-15T04:57:09.028513879+07:00 I0914 21:57:09.028447       1 client.go:80] LoxiHealthCheckChan: loxilb(http://172.22.4.111:11111) is alive
2023-09-15T04:57:09.127112605+07:00 I0914 21:57:09.126944       1 shared_informer.go:280] Caches are synced for LoxilbLoadBalancerManager
2023-09-15T04:59:02.492264943+07:00 I0914 21:59:02.490966       1 ippool.go:81] Allocate ServiceIP 172.22.4.0:5672 (tcp)
2023-09-15T04:59:02.492303065+07:00 I0914 21:59:02.491072       1 loadbalancer.go:573] Endpoint IP Pairs [172.22.4.106 172.22.4.105 172.22.4.108 172.22.4.104 172.22.4.107 172.22.4.109]
2023-09-15T04:59:02.492311360+07:00 I0914 21:59:02.491083       1 loadbalancer.go:574] Secondary IP Pairs []
2023-09-15T04:59:02.494510286+07:00 I0914 21:59:02.493900       1 loadbalancer.go:668] load-balancer ([{true false  {{172.22.4.0 5672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.106 31022 2 } {172.22.4.105 31022 2 } {172.22.4.108 31022 2 } {172.22.4.104 31022 2 } {172.22.4.107 31022 1 } {172.22.4.109 31022 1 }]}}]) added
2023-09-15T04:59:02.497754214+07:00 I0914 21:59:02.497049       1 loadbalancer.go:668] load-balancer ([{false true  {{172.22.4.0 15672 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.106 32251 2 } {172.22.4.105 32251 2 } {172.22.4.108 32251 2 } {172.22.4.104 32251 2 } {172.22.4.107 32251 1 } {172.22.4.109 32251 1 }]}}]) added
2023-09-15T04:59:02.505959877+07:00 I0914 21:59:02.505087       1 loadbalancer.go:668] load-balancer ([{false true  {{172.22.4.0 15692 tcp 0 2 false true 1800 0 true  0  } [] [{172.22.4.106 31570 2 } {172.22.4.105 31570 2 } {172.22.4.108 31570 2 } {172.22.4.104 31570 2 } {172.22.

172.22.4.110

root@dr-worker-007:/# loxicmd get lb -o wide
| EXTERNAL IP | SECONDARY IPS | PORT  | PROTOCOL | BLOCK | SELECT |  MODE   | ENDPOINT IP  | TARGET PORT | WEIGHT | STATE  |
|-------------|---------------|-------|----------|-------|--------|---------|--------------|-------------|--------|--------|
| 172.22.4.0  |               |  5672 | tcp      |     0 | rr     | fullnat | 172.22.4.104 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31022 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31022 |      1 | active |
| 172.22.4.0  |               | 15672 | tcp      |     0 | rr     | fullnat | 172.22.4.104 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       32251 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       32251 |      1 | active |
| 172.22.4.0  |               | 15692 | tcp      |     0 | rr     | fullnat | 172.22.4.104 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31570 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31570 |      1 | active |
root@dr-worker-007:/#

2023-09-14 21:59:02 suitable source for 172.22.4.104: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.105: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.106: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.107: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.108: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.109: 172.22.4.0
2023-09-14 21:59:02 [API] Load balancer POST API called. url : /netlox/v1/config/loadbalancer
2023-09-14 21:59:02 [API] lbRules : {{172.22.4.0 15692 tcp 0 0 false true 2 1800 true  0  } [] [{172.22.4.106 31570 2 } {172.22.4.105 31570 2 } {172.22.4.108 31570 2 } {172.22.4.104 31570 2 } {172.22.4.107 31570 1 } {172.22.4.109 31570 1 }]}
2023-09-14 21:59:02 ep-host added 172.22.4.104_tcp_31570:0
2023-09-14 21:59:02 ep-host added 172.22.4.105_tcp_31570:1
2023-09-14 21:59:02 ep-host added 172.22.4.106_tcp_31570:2
2023-09-14 21:59:02 ep-host added 172.22.4.107_tcp_31570:3
2023-09-14 21:59:02 ep-host added 172.22.4.108_tcp_31570:0
2023-09-14 21:59:02 ep-host added 172.22.4.109_tcp_31570:1
2023-09-14 21:59:02 nat lb-rule added - 3:dst-172.22.4.0/32,proto-6,dport-15692,-do-fullnat:eip-172.22.4.104,ep-31570,w-2,alive|eip-172.22.4.105,ep-31570,w-2,alive|eip-172.22.4.106,ep-31570,w-2,alive|eip-172.22.4.107,ep-31570,w-1,alive|eip-172.22.4.108,ep-31570,w-2,alive|eip-172.22.4.109,ep-31570,w-1,alive|
2023-09-14 21:59:02 suitable source for 172.22.4.104: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.105: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.106: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.107: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.108: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.109: 172.22.4.0
2023-09-14 21:59:02 inactive ep - 172.22.4.106_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.105_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.107_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.106_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.105_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.107_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.106_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.109_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.104_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.109_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.105_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.104_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.108_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.107_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.108_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.109_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.104_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.108_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:03 [DP] LB rule 172.22.4.0 add[OK]
2023-09-14 21:59:03 [DP] LB rule 172.22.4.0 add[OK]
2023-09-14 21:59:03 [DP] LB rule 172.22.4.0 add[OK]
2023-09-14 21:59:03 neigh rtpair - 172.22.4.107/32->172.22.4.107
2023-09-14 21:59:03 rt added - 172.22.4.107/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 7 223 103] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.107:root (12)
2023-09-14 21:59:03 neigh rtpair - 172.22.4.106/32->172.22.4.106
2023-09-14 21:59:03 rt added - 172.22.4.106/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 55 122 36] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.106:root (13)
2023-09-14 21:59:03 neigh rtpair - 172.22.4.104/32->172.22.4.104
2023-09-14 21:59:03 rt added - 172.22.4.104/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 166 237 28] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.104:root (14)
2023-09-14 21:59:03 neigh rtpair - 172.22.4.109/32->172.22.4.109
2023-09-14 21:59:03 rt added - 172.22.4.109/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 154 29 116] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.109:root (15)
2023-09-14 21:59:03 neigh rtpair - 172.22.4.108/32->172.22.4.108
2023-09-14 21:59:03 rt added - 172.22.4.108/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 88 85 237] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.108:root (16)
2023-09-14 21:59:04 active ep - 172.22.4.107_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:04 active ep - 172.22.4.109_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:04 active ep - 172.22.4.106_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:04 active ep - 172.22.4.106_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:04 active ep - 172.22.4.105_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:04 active ep - 172.22.4.104_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:04 active ep - 172.22.4.105_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:04 active ep - 172.22.4.104_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:04 active ep - 172.22.4.109_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:04 active ep - 172.22.4.107_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:04 active ep - 172.22.4.108_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:04 active ep - 172.22.4.109_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:04 active ep - 172.22.4.108_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:04 active ep - 172.22.4.107_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:04 active ep - 172.22.4.104_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:04 active ep - 172.22.4.108_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:04 active ep - 172.22.4.105_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:04 active ep - 172.22.4.106_tcp_31570:tcp:31570(0s)
21:59:12 TRACE loxilb_libdp.c:1978: ct: #172.22.4.108:0 -> 172.22.4.110:0 (1)# rid:0 est:0 nat:0 (Aged:10022319670ns:0:0)
21:59:12 TRACE loxilb_libdp.c:1978: ct: #172.22.4.105:0 -> 172.22.4.110:0 (1)# rid:0 est:0 nat:0 (Aged:10027886241ns:0:0)
21:59:12 TRACE loxilb_libdp.c:1978: ct: #172.22.4.106:0 -> 172.22.4.110:0 (1)# rid:0 est:0 nat:0 (Aged:10031389756ns:0:0)
21:59:12 TRACE loxilb_libdp.c:1978: ct: #172.22.4.104:0 -> 172.22.4.110:0 (1)# rid:0 est:0 nat:0 (Aged:10024436146ns:0:0)
21:59:12 TRACE loxilb_libdp.c:1978: ct: #172.22.4.107:0 -> 172.22.4.110:0 (1)# rid:0 est:0 nat:0 (Aged:10025548320ns:0:0)
21:59:12 TRACE loxilb_libdp.c:1978: ct: #172.22.4.109:0 -> 172.22.4.110:0 (1)# rid:0 est:0 nat:0 (Aged:10024714948ns:0:0)
2023-09-14 22:04:42 [API] Load balancer GET API called. url : /netlox/v1/config/loadbalancer/all

172.22.4.111

root@dr-worker-008:/# loxicmd get lb -o wide
| EXTERNAL IP | SECONDARY IPS | PORT  | PROTOCOL | BLOCK | SELECT |  MODE   | ENDPOINT IP  | TARGET PORT | WEIGHT | STATE  |
|-------------|---------------|-------|----------|-------|--------|---------|--------------|-------------|--------|--------|
| 172.22.4.0  |               |  5672 | tcp      |     0 | rr     | fullnat | 172.22.4.104 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31022 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31022 |      1 | active |
| 172.22.4.0  |               | 15672 | tcp      |     0 | rr     | fullnat | 172.22.4.104 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       32251 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       32251 |      1 | active |
| 172.22.4.0  |               | 15692 | tcp      |     0 | rr     | fullnat | 172.22.4.104 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31570 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31570 |      1 | active |
root@dr-worker-008:/#

2023-09-14 21:53:01 ebpf intfmap added - 5 vlan 0 -> 1
2023-09-14 21:53:01 ebpf txintfmap added - 1 -> 5
2023-09-14 21:53:01 ebpf intfmap added - 6 vlan 0 -> 511
2023-09-14 21:53:01 ebpf txintfmap added - 511 -> 6
2023-09-14 21:53:05 Get xsync()
2023-09-14 21:57:09 neigh rtpair - 172.22.4.105/32->172.22.4.105
2023-09-14 21:57:09 rt added - 172.22.4.105/32:root
2023-09-14 21:57:09 added fdb ent, {[0 12 41 66 220 175] 3804}
2023-09-14 21:57:09 neigh added - 172.22.4.105:root (11)
2023-09-14 21:59:02 [API] Load balancer POST API called. url : /netlox/v1/config/loadbalancer
2023-09-14 21:59:02 [API] lbRules : {{172.22.4.0 5672 tcp 0 0 false true 2 1800 true  0  } [] [{172.22.4.106 31022 2 } {172.22.4.105 31022 2 } {172.22.4.108 31022 2 } {172.22.4.104 31022 2 } {172.22.4.107 31022 1 } {172.22.4.109 31022 1 }]}
2023-09-14 21:59:02 ep-host added 172.22.4.104_tcp_31022:0
2023-09-14 21:59:02 ep-host added 172.22.4.105_tcp_31022:1
2023-09-14 21:59:02 ep-host added 172.22.4.106_tcp_31022:2
2023-09-14 21:59:02 ep-host added 172.22.4.107_tcp_31022:3
2023-09-14 21:59:02 ep-host added 172.22.4.108_tcp_31022:0
2023-09-14 21:59:02 ep-host added 172.22.4.109_tcp_31022:1
2023-09-14 21:59:02 nat lb-rule added - 1:dst-172.22.4.0/32,proto-6,dport-5672,-do-fullnat:eip-172.22.4.104,ep-31022,w-2,alive|eip-172.22.4.105,ep-31022,w-2,alive|eip-172.22.4.106,ep-31022,w-2,alive|eip-172.22.4.107,ep-31022,w-1,alive|eip-172.22.4.108,ep-31022,w-2,alive|eip-172.22.4.109,ep-31022,w-1,alive|
2023-09-14 21:59:02 suitable source for 172.22.4.104: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.105: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.106: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.107: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.108: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.109: 172.22.4.0
2023-09-14 21:59:02 [API] Load balancer POST API called. url : /netlox/v1/config/loadbalancer
2023-09-14 21:59:02 [API] lbRules : {{172.22.4.0 15672 tcp 0 0 false true 2 1800 true  0  } [] [{172.22.4.106 32251 2 } {172.22.4.105 32251 2 } {172.22.4.108 32251 2 } {172.22.4.104 32251 2 } {172.22.4.107 32251 1 } {172.22.4.109 32251 1 }]}
2023-09-14 21:59:02 ep-host added 172.22.4.104_tcp_32251:2
2023-09-14 21:59:02 ep-host added 172.22.4.105_tcp_32251:3
2023-09-14 21:59:02 ep-host added 172.22.4.106_tcp_32251:0
2023-09-14 21:59:02 ep-host added 172.22.4.107_tcp_32251:1
2023-09-14 21:59:02 ep-host added 172.22.4.108_tcp_32251:2
2023-09-14 21:59:02 ep-host added 172.22.4.109_tcp_32251:3
2023-09-14 21:59:02 nat lb-rule added - 2:dst-172.22.4.0/32,proto-6,dport-15672,-do-fullnat:eip-172.22.4.104,ep-32251,w-2,alive|eip-172.22.4.105,ep-32251,w-2,alive|eip-172.22.4.106,ep-32251,w-2,alive|eip-172.22.4.107,ep-32251,w-1,alive|eip-172.22.4.108,ep-32251,w-2,alive|eip-172.22.4.109,ep-32251,w-1,alive|
2023-09-14 21:59:02 suitable source for 172.22.4.104: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.105: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.106: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.107: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.108: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.109: 172.22.4.0
2023-09-14 21:59:02 [API] Load balancer POST API called. url : /netlox/v1/config/loadbalancer
2023-09-14 21:59:02 [API] lbRules : {{172.22.4.0 15692 tcp 0 0 false true 2 1800 true  0  } [] [{172.22.4.106 31570 2 } {172.22.4.105 31570 2 } {172.22.4.108 31570 2 } {172.22.4.104 31570 2 } {172.22.4.107 31570 1 } {172.22.4.109 31570 1 }]}
2023-09-14 21:59:02 ep-host added 172.22.4.104_tcp_31570:0
2023-09-14 21:59:02 ep-host added 172.22.4.105_tcp_31570:1
2023-09-14 21:59:02 ep-host added 172.22.4.106_tcp_31570:2
2023-09-14 21:59:02 ep-host added 172.22.4.107_tcp_31570:3
2023-09-14 21:59:02 ep-host added 172.22.4.108_tcp_31570:0
2023-09-14 21:59:02 ep-host added 172.22.4.109_tcp_31570:1
2023-09-14 21:59:02 nat lb-rule added - 3:dst-172.22.4.0/32,proto-6,dport-15692,-do-fullnat:eip-172.22.4.104,ep-31570,w-2,alive|eip-172.22.4.105,ep-31570,w-2,alive|eip-172.22.4.106,ep-31570,w-2,alive|eip-172.22.4.107,ep-31570,w-1,alive|eip-172.22.4.108,ep-31570,w-2,alive|eip-172.22.4.109,ep-31570,w-1,alive|
2023-09-14 21:59:02 suitable source for 172.22.4.104: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.105: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.106: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.107: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.108: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.109: 172.22.4.0
2023-09-14 21:59:02 [DP] LB rule 172.22.4.0 add[OK]
2023-09-14 21:59:02 [DP] LB rule 172.22.4.0 add[OK]
2023-09-14 21:59:02 [DP] LB rule 172.22.4.0 add[OK]
2023-09-14 21:59:03 neigh rtpair - 172.22.4.106/32->172.22.4.106
2023-09-14 21:59:03 rt added - 172.22.4.106/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 55 122 36] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.106:root (12)
2023-09-14 21:59:03 neigh rtpair - 172.22.4.108/32->172.22.4.108
2023-09-14 21:59:03 rt added - 172.22.4.108/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 88 85 237] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.108:root (13)
2023-09-14 21:59:03 neigh rtpair - 172.22.4.107/32->172.22.4.107
2023-09-14 21:59:03 rt added - 172.22.4.107/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 7 223 103] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.107:root (14)
2023-09-14 21:59:03 neigh rtpair - 172.22.4.109/32->172.22.4.109
2023-09-14 21:59:03 rt added - 172.22.4.109/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 154 29 116] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.109:root (15)
2023-09-14 21:59:03 neigh rtpair - 172.22.4.104/32->172.22.4.104
2023-09-14 21:59:03 rt added - 172.22.4.104/32:root
2023-09-14 21:59:03 added fdb ent, {[0 12 41 166 237 28] 3804}
2023-09-14 21:59:03 neigh added - 172.22.4.104:root (16)
2023-09-14 22:09:24 [API] Load balancer GET API called. url : /netlox/v1/config/loadbalancer/all

172.22.4.112

root@dr-worker-009:/# loxicmd get lb -o wide
| EXTERNAL IP | SECONDARY IPS | PORT  | PROTOCOL | BLOCK | SELECT |  MODE   | ENDPOINT IP  | TARGET PORT | WEIGHT | STATE  |
|-------------|---------------|-------|----------|-------|--------|---------|--------------|-------------|--------|--------|
| 172.22.4.0  |               |  5672 | tcp      |     0 | rr     | fullnat | 172.22.4.104 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31022 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31022 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31022 |      1 | active |
| 172.22.4.0  |               | 15672 | tcp      |     0 | rr     | fullnat | 172.22.4.104 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       32251 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       32251 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       32251 |      1 | active |
| 172.22.4.0  |               | 15692 | tcp      |     0 | rr     | fullnat | 172.22.4.104 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.105 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.106 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.107 |       31570 |      1 | active |
|             |               |       |          |       |        |         | 172.22.4.108 |       31570 |      2 | active |
|             |               |       |          |       |        |         | 172.22.4.109 |       31570 |      1 | active |
root@dr-worker-009:/#

2023-09-14 21:59:02 ep-host added 172.22.4.104_tcp_32251:2
2023-09-14 21:59:02 ep-host added 172.22.4.105_tcp_32251:3
2023-09-14 21:59:02 ep-host added 172.22.4.106_tcp_32251:0
2023-09-14 21:59:02 ep-host added 172.22.4.107_tcp_32251:1
2023-09-14 21:59:02 ep-host added 172.22.4.108_tcp_32251:2
2023-09-14 21:59:02 ep-host added 172.22.4.109_tcp_32251:3
2023-09-14 21:59:02 nat lb-rule added - 2:dst-172.22.4.0/32,proto-6,dport-15672,-do-fullnat:eip-172.22.4.104,ep-32251,w-2,alive|eip-172.22.4.105,ep-32251,w-2,alive|eip-172.22.4.106,ep-32251,w-2,alive|eip-172.22.4.107,ep-32251,w-1,alive|eip-172.22.4.108,ep-32251,w-2,alive|eip-172.22.4.109,ep-32251,w-1,alive|
2023-09-14 21:59:02 suitable source for 172.22.4.104: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.105: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.106: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.107: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.108: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.109: 172.22.4.0
2023-09-14 21:59:02 [API] Load balancer POST API called. url : /netlox/v1/config/loadbalancer
2023-09-14 21:59:02 [API] lbRules : {{172.22.4.0 15692 tcp 0 0 false true 2 1800 true  0  } [] [{172.22.4.106 31570 2 } {172.22.4.105 31570 2 } {172.22.4.108 31570 2 } {172.22.4.104 31570 2 } {172.22.4.107 31570 1 } {172.22.4.109 31570 1 }]}
2023-09-14 21:59:02 ep-host added 172.22.4.104_tcp_31570:0
2023-09-14 21:59:02 ep-host added 172.22.4.105_tcp_31570:1
2023-09-14 21:59:02 ep-host added 172.22.4.106_tcp_31570:2
2023-09-14 21:59:02 ep-host added 172.22.4.107_tcp_31570:3
2023-09-14 21:59:02 ep-host added 172.22.4.108_tcp_31570:0
2023-09-14 21:59:02 ep-host added 172.22.4.109_tcp_31570:1
2023-09-14 21:59:02 nat lb-rule added - 3:dst-172.22.4.0/32,proto-6,dport-15692,-do-fullnat:eip-172.22.4.104,ep-31570,w-2,alive|eip-172.22.4.105,ep-31570,w-2,alive|eip-172.22.4.106,ep-31570,w-2,alive|eip-172.22.4.107,ep-31570,w-1,alive|eip-172.22.4.108,ep-31570,w-2,alive|eip-172.22.4.109,ep-31570,w-1,alive|
2023-09-14 21:59:02 suitable source for 172.22.4.104: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.105: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.106: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.107: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.108: 172.22.4.0
2023-09-14 21:59:02 suitable source for 172.22.4.109: 172.22.4.0
2023-09-14 21:59:02 inactive ep - 172.22.4.107_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.106_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.104_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.105_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.109_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.106_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.109_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.107_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.105_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.107_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.105_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.108_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.108_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.108_tcp_31022:tcp:31022(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.104_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.106_tcp_32251:tcp:32251(next try after 60s)
2023-09-14 21:59:02 inactive ep - 172.22.4.104_tcp_31570:tcp:31570(next try after 60s)
2023-09-14 21:59:02 neigh rtpair - 172.22.4.107/32->172.22.4.107
2023-09-14 21:59:02 rt added - 172.22.4.107/32:root
2023-09-14 21:59:02 added fdb ent, {[0 12 41 7 223 103] 3804}
2023-09-14 21:59:02 neigh added - 172.22.4.107:root (12)
2023-09-14 21:59:02 neigh rtpair - 172.22.4.106/32->172.22.4.106
2023-09-14 21:59:02 rt added - 172.22.4.106/32:root
2023-09-14 21:59:02 added fdb ent, {[0 12 41 55 122 36] 3804}
2023-09-14 21:59:02 neigh added - 172.22.4.106:root (13)
2023-09-14 21:59:02 neigh rtpair - 172.22.4.104/32->172.22.4.104
2023-09-14 21:59:02 rt added - 172.22.4.104/32:root
2023-09-14 21:59:02 added fdb ent, {[0 12 41 166 237 28] 3804}
2023-09-14 21:59:02 neigh added - 172.22.4.104:root (14)
2023-09-14 21:59:02 neigh rtpair - 172.22.4.109/32->172.22.4.109
2023-09-14 21:59:02 rt added - 172.22.4.109/32:root
2023-09-14 21:59:02 added fdb ent, {[0 12 41 154 29 116] 3804}
2023-09-14 21:59:02 neigh added - 172.22.4.109:root (15)
2023-09-14 21:59:02 neigh rtpair - 172.22.4.108/32->172.22.4.108
2023-09-14 21:59:02 rt added - 172.22.4.108/32:root
2023-09-14 21:59:02 added fdb ent, {[0 12 41 88 85 237] 3804}
2023-09-14 21:59:02 neigh added - 172.22.4.108:root (16)
2023-09-14 21:59:03 [DP] LB rule 172.22.4.0 add[OK]
2023-09-14 21:59:03 [DP] LB rule 172.22.4.0 add[OK]
2023-09-14 21:59:03 [DP] LB rule 172.22.4.0 add[OK]
2023-09-14 21:59:06 active ep - 172.22.4.104_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:06 active ep - 172.22.4.107_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:06 active ep - 172.22.4.108_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:06 active ep - 172.22.4.107_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:06 active ep - 172.22.4.108_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:06 active ep - 172.22.4.109_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:06 active ep - 172.22.4.104_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:06 active ep - 172.22.4.105_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:06 active ep - 172.22.4.106_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:06 active ep - 172.22.4.106_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:06 active ep - 172.22.4.105_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:06 active ep - 172.22.4.104_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:06 active ep - 172.22.4.106_tcp_31570:tcp:31570(0s)
2023-09-14 21:59:06 active ep - 172.22.4.107_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:06 active ep - 172.22.4.108_tcp_31022:tcp:31022(0s)
2023-09-14 21:59:06 active ep - 172.22.4.105_tcp_32251:tcp:32251(0s)
2023-09-14 21:59:06 active ep - 172.22.4.109_tcp_31022:tcp:31022(0s)
21:59:18 TRACE loxilb_libdp.c:1978: ct: #172.22.4.105:0 -> 172.22.4.112:0 (1)# rid:0 est:0 nat:0 (Aged:16023944681ns:0:0)
21:59:18 TRACE loxilb_libdp.c:1978: ct: #172.22.4.107:0 -> 172.22.4.112:0 (1)# rid:0 est:0 nat:0 (Aged:16026384252ns:0:0)
21:59:18 TRACE loxilb_libdp.c:1978: ct: #172.22.4.106:0 -> 172.22.4.112:0 (1)# rid:0 est:0 nat:0 (Aged:16017663981ns:0:0)
21:59:18 TRACE loxilb_libdp.c:1978: ct: #172.22.4.104:0 -> 172.22.4.112:0 (1)# rid:0 est:0 nat:0 (Aged:16015535692ns:0:0)
21:59:18 TRACE loxilb_libdp.c:1978: ct: #172.22.4.108:0 -> 172.22.4.112:0 (1)# rid:0 est:0 nat:0 (Aged:16019940437ns:0:0)
21:59:18 TRACE loxilb_libdp.c:1978: ct: #172.22.4.109:0 -> 172.22.4.112:0 (1)# rid:0 est:0 nat:0 (Aged:16028914093ns:0:0)
2023-09-14 22:10:52 [API] Load balancer GET API called. url : /netlox/v1/config/loadbalancer/all

apiVersion: v1
kind: Service
metadata:
  annotations:
   # If there is a need to do liveness check from loxilb
   loxilb.io/liveness: "yes"
   # Specify LB mode - one of default, onearm or fullnat 
   loxilb.io/lbmode: "fullnat"
   # Specify loxilb IPAM mode - one of ipv4, ipv6 or ipv6to4 
   loxilb.io/ipam: "ipv4"
   # Specify number of secondary networks for multi-homing
   # Only valid for SCTP currently
   # loxilb.io/num-secondary-networks: "2
  labels:
    app.kubernetes.io/name: production-rabbitmqcluster
  name: production-rabbitmqcluster-lb1
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - appProtocol: amqp
    name: amqp
    port: 5672
    protocol: TCP
    targetPort: 5672
  - appProtocol: http
    name: management
    port: 15672
    protocol: TCP
    targetPort: 15672
  - appProtocol: prometheus.io/metrics
    name: prometheus
    port: 15692
    protocol: TCP
    targetPort: 15692
  selector:
    app.kubernetes.io/name: production-rabbitmqcluster
  sessionAffinity: None
  type: LoadBalancer
  loadBalancerClass: loxilb.io/loxilb

Hi @4ss3g4f, Thank you for reaching out to us. We would need more information to debug this. Please let us know: 1) How are running LoxiLB? In-cluster or Outside of the cluster? 2) Service with other ports are accessible?

And, we have sent you Slack channel request on your email. Kindly join. It would be faster to communicate on Slack.

How are running LoxiLB? In-cluster or Outside of the cluster? Detail below
Service with other ports are accessible? none of the ports available

We are running loxilb in flatcar linux kubernetes cluster with detail:
- k8s v1.28.1
- etcd v3.5.9
- flannel v0.22.2-amd64 with backend wireguard
- k8s cluster 3 master nodes, 6 worker nodes and 3 loxilb nodes
- loxilb running in a non worker nodes, but the loxilb nodes are connected to k8s cluster via etcd and flannel due to flatcar linux design cluster capabilities
- loxilb running as docker systemd, here is the detail:
- name: "loxilb.service" enable: true contents: | [Unit] Description=loxilb Requires=docker.service After=docker.service [Service] Restart=always ExecStartPre=-/usr/bin/docker rm loxilb ExecStart=/usr/bin/docker run --privileged \ --net=host \ --cap-add SYS_ADMIN \ -v /home/core/loxilb/log:/dev/log \ --name loxilb ghcr.io/loxilb-io/loxilb:latest ExecStop=-/usr/bin/docker stop loxilb ExecStopPost=-/usr/bin/docker rm loxilb [Install] WantedBy=multi-user.target
- and kube-loxilb.yaml:
```
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-loxilb
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kube-loxilb
rules:
```
- apiGroups:
  - "" resources:
  - nodes verbs:
  - get
  - watch
  - list
  - patch
- apiGroups:
  - "" resources:
  - pods verbs:
  - get
  - watch
  - list
  - patch
- apiGroups:
  - "" resources:
  - endpoints
  - services
  - services/status verbs:
  - get
  - watch
  - list
  - patch
  - update
- apiGroups:
  - discovery.k8s.io resources:
  - endpointslices verbs:
  - get
  - watch
  - list
- apiGroups:
  - authentication.k8s.io resources:
  - tokenreviews verbs:
  - create
- apiGroups:
  - authorization.k8s.io resources:
  - subjectaccessreviews verbs:
  - create
    
    kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: kube-loxilb roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-loxilb subjects:
- kind: ServiceAccount name: kube-loxilb namespace: kube-system
  
  apiVersion: apps/v1 kind: Deployment metadata: name: kube-loxilb namespace: kube-system labels: app: kube-loxilb-app spec: replicas: 1 selector: matchLabels: app: kube-loxilb-app template: metadata: labels: app: kube-loxilb-app spec: hostNetwork: true dnsPolicy: ClusterFirstWithHostNet tolerations:
  - effect: NoSchedule operator: Exists
    Mark the pod as a critical add-on for rescheduling.
  - key: CriticalAddonsOnly operator: Exists
  - effect: NoExecute operator: Exists priorityClassName: system-node-critical serviceAccountName: kube-loxilb terminationGracePeriodSeconds: 0 containers:
  - name: kube-loxilb image: ghcr.io/loxilb-io/kube-loxilb:latest imagePullPolicy: Always command:
  - /bin/kube-loxilb args:
  - --loxiURL=http://172.22.4.110:11111,http://172.22.4.111:11111,http://172.22.4.112:11111
  - --externalCIDR=172.22.4.0/26
    - --externalSecondaryCIDRs=124.124.124.1/24,125.125.125.1/24
  - --monitor=true
  - --setUniqueIP=true
    - --setBGP=64512
    
    - --extBGPPeers=50.50.50.1:65101,51.51.51.1:65102
    
    - --setRoles
    
    - --setLBMode=1
    
    - --config=/opt/loxilb/agent/kube-loxilb.conf
    
    resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: true capabilities: add: ["NET_ADMIN", "NET_RAW"]

So we just deploy loxilb docker via systemd and kube-loxilb.yaml file, do we need to deploy anything else ?? We hope that loxilb can replace metallb layer2/bgp, due to metallb layer2/bgp sometimes intermittent for low tps situation, We need something that will always available, hopefully.

Thank you

There were some issues with automation which has been fixed. Request you to :

Update loxilb image ( docker pull ghcr.io/loxilb-io/loxilb:latest ) in all nodes
Update to latest kube-loxilb image (ghcr.io/loxilb-io/kube-loxilb:latest) (delete and redeploy after changing following yaml option)
In kube-loxilb yaml, please add/enable the following option :
```
- --setRoles=0.0.0.0
```
Enable arp_accept in client node (optional step for better reachability)
```
$ sysctl net.ipv4.conf.all.arp_accept=1
$ sysctl net.ipv4.conf.default.arp_accept=1
```
OR on a per interface as applicable to your testbed

$ sysctl net.ipv4.conf.eth1.arp_accept=1

Our CICD is also being updated to reflect this particular scenario. Will be continuously checking for issues as well as for further updates from your sides (if any)

Now is working properly,

Loxilb team are awesome,

Thank you

Will close this one. There seems to be no problem in active-standby configuration. We will check active-active configuration as part of this issue

loxilb-io / kube-loxilb

failed to generate external IP. IP Pool is full #38

Service with other ports are accessible? none of the ports available

create

kind: ServiceAccount name: kube-loxilb namespace: kube-system

Mark the pod as a critical add-on for rescheduling.

- --externalSecondaryCIDRs=124.124.124.1/24,125.125.125.1/24

- --setBGP=64512

- --extBGPPeers=50.50.50.1:65101,51.51.51.1:65102

- --setRoles

- --setLBMode=1

- --config=/opt/loxilb/agent/kube-loxilb.conf