Benchmarking a click NF showing weird behavior

[AqsaKashaf]

Hi, I have a click NF, which is using IPfilter element to implement a stateless firewall. Then I am trying to benchmark this NF. I have a three node setup, which are directly connected as node1 -> node2 -> node3. node2 is running the NF. node1 sends packets and node2 receives it, processes it, and forwards to node2. To benchmark it, I send with increasing rate, until I overload it (which means the rcv rate > fwd rate). At node1, I use scapy (with tcpreplay) to send packets. The dstIP of all packets is of node3. The problem is, when I spoof the src IP address of packets at node1, if I spoof using just one IP address, I get a different overload rate. When I use two, I get almost the double, and when I use 3, I get different values.

For example, with single spoofed address: median overload rate is 90956.0 pps with two spoofed src addresses: median overload rate is 178948.0 pps with three spoofed src addresses: median overload rate is 200966.5 pps

I am not sure why is this happening. I am not using multithreading. My firewall code is:

  src :: FromDevice(eth4, SNIFFER false);
  src -> CheckIPHeader2(14)
        -> IPFilter(1000 rules, all packets that I send match the second last rule)
       -> Strip(14)
        -> Queue
        -> EtherEncap(0x0800,xxx, yyyy)
        -> ToDevice(eth5);