Question about environment requirements for Katran to run "load balancer on a stick"

[tantm3]

Hi everyone,

I am researching Katran to build a load balancer director cluster (high-performance load balancer working on the lower network layer) that is the first layer proxy network packet for our L7 load balancer (public) Inside the load balancer director with Katran, we have two network interfaces: 1 private and 1 public The architect is like: User connect through Public network ===> Network device =====> Private interface Katran ====> Katran forward to Public default Gateway with Mac addr ======> Public L7 Loadbalancer The flow above is working perfectly, but through the documentation from Katran, I saw a requirement said that:

katran is built with the assumption that it's going to be used in a "load balancer on a stick" scenario: where single interface would be used both for traffic "from user to L4 lb (ingress)" and "from L4 lb to L7 lb (egress)."

I wonder if my architect violates the best practice from Katran or not because in my scenario, Katran receives packets from a group private interface and then forwards the packets to a default gateway public which I think that the traffic will go through the public interface. So, I think this question could be a room for discussion the requirement above. Thanks for your time reading my question!

[frankfeir]

While Katran could be used in the case that a single interface is used for both "from user to L4 lb (ingress)" and "from L4 lb to L7 lb (egress)", it is also perfectly fine to use Katran as you do that katran receives traffic from user via one interface and forwards them to another interface where L7 load balancer is. I will update the document to reflect the same.

[frankfeir]

While Katran could be used in the case that a single interface is used for both "from user to L4 lb (ingress)" and "from L4 lb to L7 lb (egress)", it is also perfectly fine to use Katran as you do that katran receives traffic from user via one interface and forwards them to another interface where L7 load balancer is. I will update the document to reflect the same.

Hi, @tantm3, sorry but I might misunderstood the single interface statement from the katran document. After katran encapsulates the ingress packets, katran forwards traffics to L7 load balancer by returning XDP_TX. And according to https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/implementation/xdp_actions.html#xdp-tx, "The XDP_TX action result in TX bouncing the received packet-page back out the same NIC it arrived on." In that sense, the same interface is required for the traffic "from user to L4 lb" and "from L4 lb to L7 lb".

you mentioned your architect works. I am curious to learn how you managed to forward traffics to L7 load balancer via a different network interface.

[tantm3]

Hi @frankfeir,

Thanks for your response!

I think the reason why Katran could work with the different interfaces is the trunking in my network interface, the server has two IPs, but actually, it share the same physical network port I will share more detail about my architect in lab environment:

• I have 3 server Katran running in this server: com-166: ip private - 10.50.73.166, public - 49.213.85.164 Here is the member: com-167: private - 10.50.73.167, public - 49.213.85.162 also member of Katran: com-168: private - 10.50.73.168, public - 49.213.85.163
• Here is a part of ifconfig output

  3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
  link/ether 24:6e:96:34:74:c1 brd ff:ff:ff:ff:ff:ff
  altname enp1s0f1
  inet6 fe80::266e:96ff:fe34:74c1/64 scope link 
     valid_lft forever preferred_lft forever
  4: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
  link/ether 24:6e:96:34:74:c2 brd ff:ff:ff:ff:ff:ff
  altname enp1s0f2
  5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
  link/ether 24:6e:96:34:74:c3 brd ff:ff:ff:ff:ff:ff
  altname enp1s0f3
  6: eno2.573@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
  link/ether 24:6e:96:34:74:c1 brd ff:ff:ff:ff:ff:ff
  inet 10.50.73.167/24 brd 10.50.73.255 scope global eno2.573
     valid_lft forever preferred_lft forever
  inet6 fe80::266e:96ff:fe34:74c1/64 scope link 
     valid_lft forever preferred_lft forever
  12: tunl0@NONE: <NOARP> mtu 1480 qdisc noqueue state DOWN group default qlen 1000
  link/ipip 0.0.0.0 brd 0.0.0.0
  17: eno2.3399@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
  link/ether 24:6e:96:34:74:c1 brd ff:ff:ff:ff:ff:ff
  inet 49.213.85.162/28 brd 49.213.85.175 scope global eno2.3399
     valid_lft forever preferred_lft forever
  inet6 fe80::266e:96ff:fe34:74c1/64 scope link 
     valid_lft forever preferred_lft forever

• Here is the output of the mac-add in com-166

  ip n show | grep 161
  10.50.73.161 dev eno2.573 lladdr f8:bc:12:04:3f:b9 STALE
  49.213.85.161 dev eno2.3399 lladdr 58:e4:34:56:46:e0 REACHABLE

• Here is how I run Katran

  ./build/example_grpc/katran_server_grpc -balancer_prog ./deps/bpfprog/bpf/balancer.bpf.o -default_mac 58:e4:34:56:46:e0 -forwarding_cores=0 -healthchecker_prog ./deps/bpfprog/bpf/healthchecking_ipip.o -intf=eno2.573 -ipip_intf=ipip0 -ipip6_intf=ipip60 -lru_size=10000
  ./katran_goclient -A -t 49.213.85.150:80
  ./katran_goclient -a -t 49.213.85.150:80 -r 49.213.85.162
  ./katran_goclient -a -t 49.213.85.150:80 -r 49.213.85.163

  In short, the Katran server receives packets in eno2.573 interface and then Katran will bounce packets with XDP_TX back to eno2.573 interface with the mac address is the default gateway 49.213.85.161 dev eno2.3399 lladdr 58:e4:34:56:46:e0 REACHABLE As I said earlier, the network port is trunking with two vLAN networks, that why Katran can redirect traffic to the default gateway of a different interface. If I have two different physical network port and attach each port to a specific vLAN, I think Katran will not work anymore.

[frankfeir]

Thanks for the details. I see you use the mac address 58:e4:34:56:46:e0 as the default gateway mac which is filled as the destination eth addr IIUC. So essentially, it is still one single interface eno2.573 for traffic from user to L4 and traffic from L4 to L7. Even without the trunking in your network interface, you can still specify the mac address of the default gateway and it would work. Correct me if I misunderstand anything.

[tantm3]

I think it's not working without the trunking because if there are two physical interfaces (1 public, 1 private) in the Katran server, we can imagine the server has two cables attached to it. Each cable connects to a different gateway device. So, the traffic that enters the Katran's private interface and then the XDP executes the XDP_TX in the same private interface, and finally, it's transferred to the public default gateway mac-address. With the flow above, it seems like we transfer the packet from 1 cable private but output the packet in a different cable is the public one. So, I think if there is no configuration in the default gateway, the flow is not working anymore.

Hope to hear what you think about it!

[frankfeir]

sorry I was not clear. In your case, trunking is needed because user traffic enters katran server via one interface and L7 load balancers are in another vlan. What I was trying to say is if L7 load balancers are in the same vlan that the same interface(for ingress traffic) is connected to, the gateway would be reachable directly from that interface and trunking is not needed.

[tantm3]

Oh, In my case, L7 Loadbalancer is not in the same network as Katran and we forward packets from Katran to L7 Loadbalancer using the public internet. So, I am not sure what gateway mac address I fill for Katran to forward traffic. As I understand, the server has only one default gateway, so we just need to find the gateway mac-add and give it to Katran. Despite we have two or three interfaces, the routing part is done at the default gateway, so, Katran does not care about the default gateway or interface. Is that exactly the way that Katran works?

[frankfeir]

you do need to attach katran(xdp program) to an interface and since katran returns XDP_TX and traffics are bounded back out the same interface, the gateway mac address needs to be reachable from that interface. In your case, I assume the default gateway is connected to the public interface and because of trunking, it is reachable from private interface so it works.

[tantm3]

Oh, I am clear about how the trunking makes it work now, thanks for your explanation. I am not sure that if there is no trunking and we use two separate network interfaces, the default gateway is not reachable to the public interface anymore so, Katran is not working. I will do some tests in that scenario and share the results here.

[tantm3]

I will close the question here because it's clear now! Maybe we could share more information later.