Client - Server on different VMs on same ESX server

[Ajaay]

Hi,

I'm trying to setup two instances of trex, with one running as client and the other as server, on separate VMs on the same ESX node.

What I'm finding is that when I run a test it removes the interface route so traffic to it's local gateway goes down.

I can then manually add it back in which returns connectivity but it gets removed again when I run trex.

Is this use case supported? Or is it because it's trying to ARP out of the physical host interface instead of the virtual adaptor I want it to use?

Thanks

[hhaim]

Can you send more info on the setup? Driver type etc. trex_cfg file will help to understand the problem

Hanoh

On Wed, 13 Jun 2018 at 6:12 Ajaay notifications@github.com wrote:

Hi,

I'm trying to setup two instances of trex, with one running as client and the other as server, on separate VMs on the same ESX node.

What I'm finding is that when I run a test it removes the interface route so traffic to it's local gateway goes down.

I can then manually add it back in which returns connectivity but it gets removed again when I run trex.

Is this use case supported? Or is it because it's trying to ARP out of the physical host interface instead of the virtual adaptor I want it to use?

Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cisco-system-traffic-generator/trex-core/issues/116, or mute the thread https://github.com/notifications/unsubscribe-auth/AMbjvWs4pymJsHmjqyuM0kbDa2e8HHz3ks5t8OWqgaJpZM4Ul6ed .

-- Hanoh Sent from my iPhone

[ybrustin]

Hi,

You should have separate vSwitch (and interfaces) for TRex traffic: https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_trex_on_esxi

[Ajaay]

@ybrustin - Why a separate vSwitch?

I have separate interfaces, and can still access and manage the device via it's active interface. For example:

[root@opti-mag-lab-auto-svr1 v2.41]# sudo ./dpdk_setup_ports.py -s

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:0b:00.0 'VMXNET3 Ethernet Controller' if=ens192 drv=vmxnet3 unused=igb_uio,vfio-pci,uio_pci_generic
0000:13:00.0 'VMXNET3 Ethernet Controller' if=ens224 drv=vmxnet3 unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:1b:00.0 'VMXNET3 Ethernet Controller' if=ens256 drv=vmxnet3 unused=igb_uio,vfio-pci,uio_pci_generic

Other network devices
=====================
<none>

And the ifconfig:

[root@opti-mag-lab-auto-svr1 v2.41]# ifconfig
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 100.127.1.1  netmask 255.255.255.0  broadcast 100.127.1.255
        inet6 fe80::20c:29ff:fe3b:1fc9  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:3b:1f:c9  txqueuelen 1000  (Ethernet)
        RX packets 6519  bytes 1352473 (1.2 MiB)
        RX errors 0  dropped 10  overruns 0  frame 0
        TX packets 2833  bytes 500739 (489.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 100.66.32.4  netmask 255.255.255.0  broadcast 100.66.32.255
        inet6 fe80::896d:f995:30ae:68fc  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:3b:1f:d3  txqueuelen 1000  (Ethernet)
        RX packets 76868  bytes 10510206 (10.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5717  bytes 607433 (593.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens256: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:0c:29:3b:1f:dd  txqueuelen 1000  (Ethernet)
        RX packets 2782  bytes 175656 (171.5 KiB)
        RX errors 0  dropped 2  overruns 0  frame 0
        TX packets 8131  bytes 1446018 (1.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 52  bytes 5444 (5.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 52  bytes 5444 (5.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

All of these interfaces are on the same vSwitch but on different VLANs.

I want to run ens192 as a client interface towards another VM on the same vSwitch (different VLAN) through my DUT. Here's my example config file:

[root@opti-mag-lab-auto-svr1 v2.41]# cat /etc/trex_cfg.yaml
### Config file generated by dpdk_setup_ports.py ###

- port_limit: 2
  version: 2
  interfaces: ['0b:00.0', '1b:00.0']
  port_info:
      - ip: 100.127.1.1
        default_gw: 100.127.1.254
      - ip: 100.127.1.2
        default_gw: 100.127.1.254

  platform:
      master_thread_id: 0
      latency_thread_id: 1
      dual_if:
        - socket: 0
          threads: [2,3]

  generator:
      distribution:  "seq"
      clients_start: "16.0.0.1"
      clients_end:   "16.0.0.255"
      servers_start: "48.0.0.1"
      servers_end:   "48.0.0.240"

However when I run trex, it removes the local route for the 100.127.1.0/24 network, and therefore the default gateway is unreachable... Does this matter?

[ybrustin]

However when I run trex, it removes the local route for the 100.127.1.0/24 network, and therefore the default gateway is unreachable... Does this matter?

You are using IP in config file, and hence when TRex is starting it tries to resolve MAC address of the dest IP. In STF and ASTF it's fatal error, in STL (and soon interactive ASTF) the resolve can be done later in interactive way. Basically, for other side to receive the traffic you just need to send the packets with target MAC address. You can either use MAC configuration (to specify destination MAC explicitly), or use stateless mode and resolve after the TRex is up in both sides (client and server). If you don't need high performance, you can use Linux native interfaces: https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_linux_interfaces

Thanks, Yaroslav.

[Ajaay]

Something like this?

- port_limit: 2
  version: 2
  interfaces: ['0b:00.0', '1b:00.0']
  port_info:
      - dst_mac: "00:0c:29:f0:3e:bc"

  platform:
      master_thread_id: 0
      latency_thread_id: 1
      dual_if:
        - socket: 0
          threads: [2,3]

  generator:
      distribution:  "seq"
      clients_start: "16.0.0.1"
      clients_end:   "16.0.0.255"
      servers_start: "48.0.0.1"
      servers_end:   "48.0.0.240"

I currently get the following error:

[root@opti-mag-lab-auto-svr1 v2.41]# sudo ./t-rex-64 -f astf/http_simple.py -m 100 -d 100 -c 1 --astf-client-mask 0x1 -l 100 -k 10
WARNING: tried to configure 2048 hugepages for socket 0, but result is: 1679
Trying to bind to igb_uio ...
/bin/python dpdk_nic_bind.py --bind=igb_uio 0000:0b:00.0 0000:1b:00.0
The ports are bound/configured.
Starting  TRex v2.41 please wait  ...
 set driver name net_vmxnet3
 driver capability  : TSO
 Number of ports found: 2
zmq publisher at: tcp://*:4500
 wait 1 sec .
port : 0
------------
link         :  link : Link Up - speed 10000 Mbps - full-duplex
promiscuous  : 0
port : 1
------------
link         :  link : Link Up - speed 10000 Mbps - full-duplex
promiscuous  : 0
 number of ports         : 2
 max cores for 2 ports   : 1
 max queue per port      : 3
yaml-cpp: error at line 4, column 14: illegal map value

[ybrustin]

Something like this?

- port_limit: 2
  version: 2
  interfaces: ['0b:00.0', '1b:00.0']
  port_info:
      - dst_mac: "00:0c:29:f0:3e:bc"

1. The key is "dest_mac"
2. You should specify it for both ports

yaml-cpp: error at line 4, column 14: illegal map value

There is probably illegal symbol there, try looking in hexdump or uploading the file as is.

[Ajaay]

Ok, I just used the interactive mode to generate a MAC based config (attached).

trex_cfg.yaml.txt

This is still reporting the same error when I try and run it.

Thanks

[ybrustin]

By the way, the stage where the error appears, is after parsing of this config by YAML is done. There is probably error in ASTF generated (another) YAML file. Did you edit it manually? (/tmp/astf.json etc.)

[Ajaay]

No I've not edited any files other than the config file. Not knowingly at least...

is there anymore verbose logging perhaps that I could look through?

[ybrustin]

Try running with -v 7 to see more output. Another thing, try deleting the file /tmp/astf.json (Before you delete it, please attach here)

[ybrustin]

You do not specify --astf flag, and the Python file is parsed as STF YAML file.

[Ajaay]

If I want to run it in client only mode, do I need to run it with both --astf and --ast-client-mask [mask] then?

[ybrustin]

Yes.