Changing start source MAC address and DUID so that multiple instances of bngblaster can be run towards the same BNG

[mivsvit]

In order to scale test a BNG with a large number of dual-stack IPoE sessions, bngblaster may need to be run on more than one host if the CPU/network resouces of a single host are not enough. The host I am using supports up to about 25k sessions before packet loss occurs although the BNG supports up to 120k.

The BNG under test rejects sessions with duplicate MACs / DHCPv6 DUIDs so they need to be unique between bngblaster instances.

However, the src MAC address and IPv6 DUID appear to be hard coded such that two instances of bngblaster will use the same DUID/src MAC address as they interate over the session identifiers.

I have not found a way to do this by configuration option or agument when calling bngblaster. There seem to be two access interface configuration options i1_start and i2_start which I was hoping could be used to configure an offset for the session ID for multiple bngblaster instances. The documentation at

https://rtbrick.github.io/bngblaster/configuration/index.html#access-interfaces

defines i1_start and i2_start as Iterator for usage in strings {i1} and Iterator for usage in strings {i2} respectively. It's not too clear what they are to be used for. Changing them from their defaults of 1 to something else does not appear to change the MAC src address, IPv6 DUID or other DHCPv6 parameters based on the bngblaster session_id.

I have quickly tested that compiling a second bngblaster instance after changing session->client_mac[1] and session->dhcpv6_duid[3] from their defaults works.

--- a/code/bngblaster/src/bbl_session.c
+++ b/code/bngblaster/src/bbl_session.c
@@ -559,7 +559,7 @@ bbl_sessions_init(bbl_ctx_s *ctx)

         /* Set client OUI to locally administered */
         session->client_mac[0] = 0x02;
-        session->client_mac[1] = 0x00;
+        session->client_mac[1] = 0x01;
         session->client_mac[2] = 0x00;
         /* Use session identifier for remaining bytes */
         session->client_mac[3] = i>>16;
@@ -590,7 +590,7 @@ bbl_sessions_init(bbl_ctx_s *ctx)

         /* Set DHCPv6 DUID */
         session->dhcpv6_duid[1] = 3;
-        session->dhcpv6_duid[3] = 1;
+        session->dhcpv6_duid[3] = 2;
         memcpy(&session->dhcpv6_duid[4], session->client_mac, ETH_ADDR_LEN);

         /* Init string variables/iterators */

If there's not a current way of doing this in the configuration, I believe there is a good use-case to add it as a feature and in such a way that other access interface configuration options such as inner-vlan-min that increment with session identifier work as before.

Additional explanation for the configuration options i1_start and i2_start etc in the documentation would also probably be useful.

[GIC-de]

Could you share your configuration? Actually would assume that a single BNG Blaster instance is able to handle much more sessions.

Are you testing with traffic streams per session? If yes, have you tried to assign the traffic streams to TX threads?

In general, I guess I can add a configurable session-id offset here, which will also change the MAC address because the last 6 bytes of MAC address are always equal to the session-id.

[mivsvit]

Thanks very much for the useful reply. Not tried traffic-streams so far, so no streams config. Only session-traffic:

    "interfaces": {
        "qdisc-bypass": true,
        "io-mode": "packet_mmap_raw",
         "network": {
         "interface": "ens160.2016",
         "address": "172.31.19.5",
         "gateway": "172.31.19.4",
         "address-ipv6": "2001:db8:2016::2",
         "gateway-ipv6": "2001:db8:2016::1"
        },
        "access": [
        {
            "interface": "ens224f1",
            "type": "ipoe",
            "qinq": "false",
            "outer-vlan-min": 2016,
            "outer-vlan-max": 2048,
            "inner-vlan-min": 2,
            "inner-vlan-max": 4094,
            "ipv4": true,
            "ipv6": true,
            "i1_start": "500",
            "i2_start": "1000",
            "vlan-mode": "1:1"
        }
     ]
    },
        "sessions": {
        "count": 25000,
        "session-time": 0,
        "max-outstanding": 16000,
        "start-rate": 1000,
        "stop-rate": 1000
    },
        "access-line": {
        "agent-remote-id": "ABCDEFGH{session-global}",
        "agent-circuit-id": "0.0.0.0/0.0.0.0 eth 0:{session-global}"
    },
    "dhcp": {
        "broadcast": true,
        "enable": true
    },
    "dhcpv6": {
        "enable": true,
        "rapid-commit": true
    },
      "session-traffic": {
        "ipv4-pps": 1,
        "ipv6pd-pps": 1
    }

[GIC-de]

You can try with the following configuration to see if performance becomes better.

{
    "interfaces": {
        "tx-interval": 1,
        "rx-interval": 1,
        "io-slots": 4096,
    }
}

I expect also a mayor improvement as soon as we support DPDK. In general I plan to work on performance in the next weeks to be able to scale up to 10M PPS.

[mivsvit]

Thanks for the suggestions.

Adding the suggested config below and sending session-traffic does not seem to improve performance, unfortunately.

{
    "interfaces": {
        "tx-interval": 1,
        "rx-interval": 1,
        "io-slots": 4096,
    }
}

With regards to the earlier comment about traffic-streams, the documentation seems to imply 200-300kpps is achievable with one thread, 1Mpps with multiple threads.

Trying this simple traffic-stream config seems to result in traffic loss earlier than when using session-traffic mostly due to TX send failed and at about 40kpps.

        "access": [
        {
            "interface": "ens224f1",
            "type": "ipoe",
            "qinq": "false",
            "outer-vlan-min": 2016,
            "outer-vlan-max": 2048,
            "inner-vlan-min": 2,
            "inner-vlan-max": 4094,
            "ipv4": true,
            "ipv6": true,
             "stream-group-id": 1,
            "vlan-mode": "1:1"
        }
     ]
    },
        "streams": [
        {
            "stream-group-id": 1,
            "name": "S1v4",
            "type": "ipv4",
            "direction": "both",
            "length": 256,
            "threaded": true,
            "thread-group": 1,
            "pps": 1
        },
                {
            "stream-group-id": 1,
            "name": "S1v6",
            "type": "ipv6pd",
            "direction": "both",
            "length": 256,
            "threaded": true,
            "thread-group": 2,
            "pps": 1
        }        

Approx 10k dual-stack sessions with 1pps traffic:

Sessions           10000 (0 PPPoE / 10000 IPoE)
  Established       9630 [##########################################################  ]
  Outstanding          0 [                                                            ]
  Terminated           0 [                                                            ]
  DHCPv4     10000/10000 [############################################################]
  DHCPv6      9630/10000 [##########################################################  ]
  Setup Time       80904 ms
  Setup Rate      119.03 CPS (MIN: 1.06 AVG: 51.98 MAX: 126.19)
  Flapped              0

Traffic Flows Verified
  Stream     37932/40000 [#########################################################   ]

  Network Interface ( ens160.2016 )
  Tx Packets                   7499901 |  19383 PPS      41867 Kbps
  Rx Packets                   7232919 |  19074 PPS      41198 Kbps
  Tx Stream Packets            7499521 |  19382 PPS
  Rx Stream Packets            7232539 |  19073 PPS     281346 Loss
  Tx Multicast Packets               0 |      0 PPS 
Access Interface ( ens224f1 )
  Tx Packets                   7805860 |  19153 PPS      42565 Kbps
  Rx Packets                   7391646 |  19357 PPS      42430 Kbps
  Tx Stream Packets            7503485 |  19079 PPS
  Rx Stream Packets            7299483 |  19356 PPS     211846 Loss
  Rx Multicast Packets               0 |      0 PPS          0 Loss

Report:

Sessions PPPoE: 0 IPoE: 64000
Sessions established: 576/64000
DHCPv6 sessions established: 3977
Setup Time: 47102 ms
Setup Rate: 12.23 CPS (MIN: 0.51 AVG: 4.56 MAX: 12.23)
Flapped: 0

Network Interface ( ens160.2016 ):
  TX:                     83813 packets
  RX:                     79873 packets
  TX Stream:              83804 packets
  RX Stream:              78085 packets (0 loss)
  TX Multicast:               0 packets
  RX Drop Unknown:            0 packets
  TX Encode Error:            0
  RX Decode Error:            0 packets
  TX Send Failed:             0
  TX No Buffer:               0
  TX Poll Kernel:             0
  RX Poll Kernel:         27996

Access Interface ( ens224f1 ):
  TX:                    322816 packets
  RX:                    123150 packets
  TX Stream:              83838 packets
  RX Stream:              78019 packets (0 loss)

  TX Poll Kernel:             0
  RX Poll Kernel:         87046

Access Interface ( ens224f1 ):
  TX:                  14438634 packets
  RX:                  13655472 packets
  TX Stream:           14002837 packets
  RX Stream:           13504899 packets (484425 loss)
  RX Multicast:               0 packets (0 loss)
  RX Drop Unknown:           58 packets
  TX Encode Error:            0 packets
  RX Decode Error:            0 packets
  TX Send Failed:        330793
  TX No Buffer:               0
  TX Poll Kernel:             0
  RX Poll Kernel:         86355

  Access Interface Protocol Packet Stats:
    ARP    TX:      48863 RX:      48862
    PADI   TX:          0 RX:          0
    PADO   TX:          0 RX:          0
    PADR   TX:          0 RX:          0
    PADS   TX:          0 RX:          0
    PADT   TX:          0 RX:          0
    LCP    TX:          0 RX:          0
    PAP    TX:          0 RX:          0
    CHAP   TX:          0 RX:          0
    IPCP   TX:          0 RX:          0
    IP6CP  TX:          0 RX:          0
    IGMP   TX:          0 RX:          0
    ICMP   TX:          0 RX:          0
    DHCP   TX:     177544 RX:      39975
    DHCPv6 TX:     190424 RX:      38212
    ICMPv6 TX:      18966 RX:      23268
    IPv4 Fragmented       RX:          0

  Access Interface Protocol Timeout Stats:
    LCP Echo Request:          0
    LCP Request:               0
    IPCP Request:              0
    IP6CP Request:             0
    PAP:                       0
    CHAP:                      0
    DHCP Request:         127554
    DHCPv6 Request:       151842
    ICMPv6 RS:                 0

Traffic Streams:
  Verified Traffic Flows: 37932/40000
  First Sequence Number Received  MIN:        1 MAX:        1
  Flow Receive Packet Loss        MIN:        2 MAX:      124
  Flow Receive Delay (msec)       MIN:    0.000 MAX:   66.478

The above was with two thread-groups, one for v4 and one for v6pd. I guess to scale it, the sessions need to be split across more thread-groups - not sure what the easiest way of doing that is. The idea is to verify forwarding for both protocol stacks for all the sessions. Or is the conclusion to wait for DPDK support?

[GIC-de]

A significant amount of loss is caused by TX Send Failed which is currently not retried and therefore counted as a loss. I plan to add a retry queue to retry packets if send has failed to ensure that stats and loss stats are correct.

In the meantime, the number of send errors could be reduced by increasing the queue size of your interfaces.

If you execute ethtool -g it shows you currently applied and max HW queue size.

$ sudo ethtool -g ens5f1
Ring parameters for ens5f1:
Pre-set maximums:
RX:                4096
RX Mini:        0
RX Jumbo:        0
TX:                4096
Current hardware settings:
RX:                512
RX Mini:        0
RX Jumbo:        0
TX:                512

Which can then be changed with:

sudo ethtool -G ens5f1 tx 4096 rx 4096
sudo ethtool -G ens5f1 tx 4096 rx 4096

You can even change the software queue size:

sudo ip link set txqueuelen 4096 dev pci1f2
sudo ip link set txqueuelen 4096 dev pci1f3

[GIC-de]

I can also see some improvement if the interfaces used by Blaster are configured without any family so that linux kernel is not handling those packets.

This can be also archived via netplan using the following configuration for each BNG Blaster interface.

network:
version: 2
renderer: networkd
ethernets:
    eth1:
    dhcp4: no
    dhcp6: no
    link-local: []
    mtu: 9000
    eth2:
    dhcp4: no
    dhcp6: no
    link-local: []
    mtu: 9000

https://rtbrick.github.io/bngblaster/interfaces.html

[GIC-de]

Could you even share the CPU model which is used?

[mivsvit]

Thanks, hadn't tried changing the software queue size but already had TX HW queue size at the max, and no protocol family configured on the interface.

$ sudo ethtool -g ens224f1
Ring parameters for ens224f1:
Pre-set maximums:
RX:             4078
RX Mini:        0
RX Jumbo:       0
TX:             4078
Current hardware settings:
RX:             4078
RX Mini:        0
RX Jumbo:       0
TX:             4078

$ ip addr show dev ens224f1
5: ens224f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 4096
    link/ether 80:61:5f:11:ff:ff brd ff:ff:ff:ff:ff:ff
$

The suggestion to increase the software queue size improves things slightly but does not eliminate the need to run multiple instances.

It's running on a VM on an ESXi hypervisor with 16 Intel Xeon E5-2667 3.20GHz cores (32 threads).

~ vim-cmd hostsvc/hostsummary | grep cpuModel
      cpuModel = "Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz",
~ esxcli hardware cpu global get
   CPU Packages: 2
   CPU Cores: 16
   CPU Threads: 32   
~    esxtop
 2:03:35pm up 50 days 54 min, 1008 worlds, 14 VMs, 27 vCPUs; CPU load average: 0.19, 0.20, 0.19
PCPU USED(%): 5.6 6.8 4.7 8.7 6.1 5.7 5.3 6.8 6.1 4.2 7.4 2.5 7.1 3.8 5.4 6.8 4.0 0.1 0.1 110 0.7 0.5 0.5 0.3 0.4 0.5 0.7 0.3 0.4 0.2 0.2 0.0 NUMA: 5.8 7.5 AVG: 6.6
PCPU UTIL(%):  20  23  16  27  19  18  16  20  19  14  21 8.7  21  13  17  21 6.8 0.3 0.3 100 2.4 1.5 1.6 0.9 1.1 1.6 2.1 0.9 1.2 0.9 0.5 0.1 NUMA:  18 7.6 AVG:  13
CORE UTIL(%):  35      37      33      33      29      27      31      33     7.0     100     3.7     2.4     2.7     2.9     2.0     nan     NUMA:  32  15 AVG:  23

        ID        GID NAME             NWLD   %USED    %RUN    %SYS   %WAIT %VMWAIT    %RDY   %IDLE  %OVRLP   %CSTP  %MLMTD  %SWPWT
     44663      44663 bngblaster        17   77.50  225.54    0.07 1489.35    0.00    3.03  598.63    0.33    0.00    0.00    0.00

[GIC-de]

I added a new configuration option mac-modifier, which allows changing the third byte of the automatically generated access session MAC address. This should suit your use case.

Pull Request: https://github.com/rtbrick/bngblaster/pull/75

{
    "interfaces": {
        "mac-modifier": 1
    }
}

Could you please verify this before merging into the dev branch?

You can build manually from mac-modifier branch or use the following dev package: https://github.com/rtbrick/bngblaster/actions/runs/2165874892

[mivsvit]

Many thanks for adding the mac-modifier config option.

It works for my use-case.

[GIC-de]

This issue is fixed in version 0.7.2.

Please verify and close the issue if the solution is accepted!

[mivsvit]

mac-modifer feature working fine in version 0.7.2. Many thanks for adding this.