UDP Round Robin Only going to single server

[DefenceLogic]

Dear All,

I am trying to setup UDP round robin load balancing for a SIEM application.

I have seesaw installed and working, I can see packets (using tcpdump on both vm's) going from the vip of 172.16.4.165 to the second server dl-clust-02.

Expected Behaviour Each udp packet send to the VServer vip (172.16.4.165) from my logging endpoint would be sent to each server in turn i.e dl-clust-01 then dl-clust-02 then back to dl-clust-01 and so on.

Actual Behaviour UDP packets are only sent to the dl-clust-02.

seesaw.cfg

`[cluster] anycast_enabled = false name = defencelogic-lb node_ipv4 = 172.16.4.163 peer_ipv4 = 172.16.4.164 vip_ipv4 = 172.16.4.160

[config_server] primary = lb1. secondary = lb2.

[interface] node = ens192 lb = ens160`

cluster.pb `seesaw_vip: < fqdn: "logger.." ipv4: "172.16.4.160/24" status: PRODUCTION

node: < fqdn: "lb1.." ipv4: "172.16.4.163/24" status: PRODUCTION

node: < fqdn: "lb2.." ipv4: "172.16.4.164/24" status: PRODUCTION

vserver: < name: "logsvr." entry_address: < fqdn: "logsvr.." ipv4: "172.16.4.165/24" status: PRODUCTION

rp: "ad1@" vserver_entry: < protocol: UDP port: 12201 scheduler: RR server_low_watermark: 0.3 healthcheck: < type: ICMP_PING interval: 5 timeout: 3 retries: 1

backend: < host: < fqdn: "dl-clust-01.." ipv4: "172.16.4.61/24" status: PRODUCTION

weight: 1

backend: < host: < fqdn: "dl-clust-02.." ipv4: "172.16.4.62/24" status: PRODUCTION

weight: 1

`

Any helps appreciated.

[liuyuan10]

could you paste what ipvsadm reports?

sudo ipvsadm -Ln

[DefenceLogic]

Hi Yuan,

I ran a test using just netcat as shown below to run five requests

echo {"version": "2.1","host":"Request - 4 -test.test.local","short_message":"Helo from seesaw","full_message":"Request 4 here\n\n Load Balancer Testing","level":1,"_user_id":9001,"_user_id":"ped","_some_env_var":"bar"} | ncat -u -w 1 logsvr.<domain fqdn>l 12201

ipvsadm output

admin@lb1:/var/log/seesaw$ sudo ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
UDP  172.16.10.165:12201 rr
  -> 172.16.10.61:12201            Route   1      0          3
  -> 172.16.10.62:12201            Route   1      0          2
dadmin@lb1:/var/log/seesaw$

I can confirm that test message are routing in a round robin style.

The problem is application related. Our logging agent sends via UDP to the VServer VIP and it seems to lock to a single SIEM back end.

When i send single JSON request as above it works perfectly, the requests are unaltered as I am running tcpdump on backend servers to check for that.

Any ideas why?

[liuyuan10]

It's because connections are tracked and your logging agent is always using the same client side udp port. you can see the connections on the seesaw instance:

sudo ipvsadm -Lnc

[4a6f656c]

Any ideas why?

By default IPVS will track flows and persist each flow (source IP, destination IP, protocol, source port, destination port) to a single backend - often spreading a flow across multiple backends will result in unwanted behaviour/problems (in the case of UDP think video streaming, VoIP calls, etc). As noted, it is likely that your application is using the same source port for all traffic, resulting in what appears to be a single flow (hence being sent to a single backend).

In cases where it is known that the UDP traffic is per packet and that you can safely send each UDP datagram to a different backend (think DNS requests), you can enable "one packet scheduling" (or OPS). This can be enabled on a Seesaw vserver_entry by adding one_packet: true to the configuration.

[DefenceLogic]

For our use case, we are using in front of our SIEM solution to provide log balancing for SIEM boxes behind.

As each agent send UDP or log source to the load balancer, the behaviour you described suits my use case. Also with firewallslogs, this is a s a requirement for us as it would mean that we could have large volume of firewall logs sending to a single backend.

If I dont use the one packet option, is there a load balancing protocol to use the least saturated backend server with UDP?

Cheers

[DefenceLogic]

I have this working now.

I will post a blog post about how I got it working for others here shortly.

[DefenceLogic]

I have enabled one_packet:true in my vserver_entry as per #102 my log sources for my zeek box still will not round robin correctly.

The output of ipvsadm -L -stats shows the following

sudo ipvsadm -L --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
UDP  logsvr.test.local:12205           0      116        0    93117        0
  -> clust-01.test.local:122        0       50        0    40502        0
  -> clust-02.test.local:122        0       66        0    52615        0
UDP  172.16.4.166:12210                  0      388        0   209490        0
  -> clust-01.test.local:122        0      388        0   209490        0
  -> clust-02.test.local:122        0        0        0        0        0

The zeek logs are on port 12210 which are being routed to clust-01.test.local whilst other logs are on 12205 which appear to balance correctly.

Relevant vserver_entry

vserver: <
  name: "zeek.test.local"
  entry_address: <
    fqdn: "logsvr.test.local."
    ipv4: "172.16.10.166/24"
    status: PRODUCTION
   >
  rp: "admin@test.local"
  vserver_entry: <
    protocol: UDP
    one_packet: true
    port: 12210
    scheduler: RR
    mode: NAT
    healthcheck: <
      type: TCP
      port: 32999
      tls_verify: false
      interval: 10
      timeout: 5
      retries: 3
    >
  >
  backend: <
    host: <
      fqdn: "clust-01.test.local."
      ipv4: "172.16.10.21/24"
      status: PRODUCTION
    >
    weight: 1
  >
  backend: <
    host: <
      fqdn: "clust-02.test.local."
      ipv4: "172.16.10.22/24"
      status: PRODUCTION
    >
    weight: 1
  >
>

I thought that once one_packet: true is enabled that IPVS would not track client source UDP ports.

Any ideas why?