Sockperf Performance over mesh testing

[tricksterisback]

I have a setup of 66 vms where i want to measure latency between each vm by pairing them in n(n-1)/2 pairs. Now i am using currently 8 ports on the server and at a time 65 clients will connect to any of the 8 ports. This has been working well for a 19 VM Sockperf mesh Test setup but when i scale it up higher number i start to see Cpu and Memory bandwidth issues. I have a 8vcore 64 gib. Has anybody done this before or is sockperf capable of this scale? Any help would be much appreciated

Here is the ansible code to deploy the setup

- name: Install sockperf server systemd
  become: true
  shell: |
    server_vm_name=$(hostname)
    server_vm_ip=$(hostname -I | awk '{print $1}')

    # Define the port file path
    port_file="/etc/default/networktestsv1/ansible/hosts/sockperf_server_ports.txt"

    # Remove the port file if it exists
    [ -f "$port_file" ] && rm "$port_file"

    # Generate the port file content
    for port in {12345..12352}; do
      echo "T:${server_vm_ip}:${port}" >> /etc/default/networktestsv1/ansible/hosts/sockperf_server_ports.txt
    done

    # Define the server command
    server_cmd="sockperf sr -f ${port_file} --threads-num 24"

    # Run the configure server service script
    bash /etc/default/networktestsv1/ansible/Scripts/configure_server_service.sh "${server_cmd}" "sockperf"
  register: configure_results
  ignore_errors: false

• name: Configure Sockperf Client on VMs hosts: localhost tasks:

  Step 1: Check the structure of the hosts

  • name: Debug group 'all' to ensure hosts are populated debug: var: groups['all']

  Step 2: Generate pairs of VMs and validate the output

  • name: Generate pairs of VMs set_fact: vm_pairs: "{{ groups['all'] | combinations(2) }}"

  • name: Debug VM pairs debug: var: vm_pairs

  Step 3: Test pairing logic before applying delegate_to

  • name: Print VM pairings loop: "{{ vm_pairs }}" debug: msg: "Pairing {{ item.0 }} as client with {{ item.1 }} as server"

• name: Configure sockperf Client systemd hosts: localhost become: true

  tasks:

  • name: Install sockperf Client systemd

    Step 4: Check if both item.0 and item.1 are defined

    delegate_to: "{{ item.0 }}" shell: | fqdn=$(getent hosts "{{ item.1 }}" | awk '{print $2}') hostname=${fqdn%%.*} server_ip=$(getent ahosts "$fqdn" | awk '/^[^ ]/{print $1; exit}')

    port=$((12345 + ({{ index }} % 8))) echo $port

    server_cmd="sockperf ping-pong -i $server_ip -m 14 -t 120 --tcp -p ${port} --full-rtt" test_tag=$hostname

    echo $server_cmd

    chmod +x /etc/default/networktestsv1/ansible/Scripts/configure_client_service.sh sh /etc/default/networktestsv1/ansible/Scripts/configure_client_service.sh "$server_cmd" "$test_tag" loop: "{{ vm_pairs }}" loop_control: index_var: index when: item.0 is defined and item.1 is defined register: script_output