MoonGen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control.
Hello! I am trying to use MoonGen to measure the latency of an experimental network. In fact I want to run this packet generator in a VM, as I have read and saw different issues here, I know that do it in virtualized NICs is not the best idea but I really need it.
I have solve some problems related to this, the first is the number of rx queues. If is useful for someone, first you have to enable multiqueue in the Host using "ethtool":
ethtool -l #e.g. eth0 --> Whit this you can see the number of queues that your NIC support
ethtool -L rx 3 --> Whit this you set 3 queues for rx as is needed to run MoonGen
Then in KVM you have to edit the XML of the VM adding this on the virtual NICs:
But now I have the problem that the virtual NICs does not support timestamping, I think that is due to de drivers ("virtio"). I don´t Know if there is any solution to this. Also I have though to use the virtual driver e1000e of intel, but with this i can´t set up the multiqueue.
Any solution? This is the output of executing the "l3-load-latency" example:
root@torre:/home/torre/MoonGen# ./build/MoonGen examples/l3-load-latency.lua 0 1[INFO] Initializing DPDK. This will take a few seconds...
EAL: Detected 4 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:01:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 1af4:1041 net_virtio
EAL: PCI device 0000:07:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 1af4:1041 net_virtio
EAL: using IOMMU type 8 (No-IOMMU)
EAL: PCI device 0000:08:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 1af4:1041 net_virtio
[INFO] Found 2 usable devices:
Device 0: 52:54:00:CF:53:8E (Red Hat, Inc Virtio network device)
Device 1: 52:54:00:E2:C7:58 (Red Hat, Inc Virtio network device)
[INFO] Waiting for devices to come up...
[INFO] Device 1 (52:54:00:E2:C7:58) is up: 10000 MBit/s
[INFO] Device 0 (52:54:00:CF:53:8E) is up: 10000 MBit/s
[INFO] 2 devices are up.
[WARN] Per-queue rate limit is not supported on this device, setting per-device rate limit to 9999 Mbit/s instead (note: this may fail as well if the NIC doesn't support any rate limiting).
[WARN] global rate limiting is not supported by the hardware or driver
[INFO] Performing ARP lookup on 10.1.0.10
[INFO] Performing ARP lookup on 10.1.0.10
[WARN] l2 filter error: Operation not supported
[WARN] l2 filter error: Operation not supported
[INFO] Destination mac: 52:54:00:e2:c7:58
[WARN] Packet size 60 is smaller than minimum timestamp size 84. Timestamped packets will be larger than load packets.
[INFO] Destination mac: 52:54:00:e2:c7:58
[FATAL] Lua error in task timerSlave
/home/torre/MoonGen/build/../libmoon/lua/device.lua:477: tx timestamping is not supported by the hardware or driver
Stack Traceback
###############
(2) Lua function 'log' at file '/home/torre/MoonGen/build/../libmoon/lua/log.lua:129' (best guess)
Local variables:
self = table: 0x411c7f98 {DEBUG:0, fatal:function: 0x408c4f70, writeToLog:function: 0x408c4f50, INFO:1 (more...)}
str = string: "tx timestamping is not supported by the hardware or driver"
(3) Lua method 'unsupported' at file '/home/torre/MoonGen/build/../libmoon/lua/device.lua:477'
Local variables:
self = [Device: id=0] {initialized:true, rxQueues:table: 0x413b1790, txQueues:table: 0x413b17e0 (more...)}
operation = string: "tx timestamping"
level = string: "fatal"
(4) Lua method 'enableTxTimestamps' at file '/home/torre/MoonGen/build/../libmoon/lua/device.lua:502'
Local variables:
self = [Device: id=0] {initialized:true, rxQueues:table: 0x413b1790, txQueues:table: 0x413b17e0 (more...)}
queue = [TxQueue: id=0, qid=1] {qid:1, dev:[Device: id=0], id:0}
(5) Lua method 'enableTimestamps' at file '/home/torre/MoonGen/build/../libmoon/lua/device.lua:577'
Local variables:
self = [TxQueue: id=0, qid=1] {qid:1, dev:[Device: id=0], id:0}
(6) Lua method 'newUdpTimestamper' at file '/home/torre/MoonGen/build/../libmoon/lua/timestamping.lua:34'
Local variables:
self = table: 0x41fea070 {newUdpTimestamper:function: 0x41fd3ee8, syncClocks:function: 0x41fd6278 (more...)}
txQueue = [TxQueue: id=0, qid=1] {qid:1, dev:[Device: id=0], id:0}
rxQueue = [RxQueue: id=1, qid=1] {qid:1, dev:[Device: id=1], id:1}
udp = boolean: true
doNotConfigureUdpPort = nil
(7) Lua function 'timerSlave' at file 'examples/l3-load-latency.lua:113' (best guess)
Local variables:
txQueue = [TxQueue: id=0, qid=1] {qid:1, dev:[Device: id=0], id:0}
rxQueue = [RxQueue: id=1, qid=1] {qid:1, dev:[Device: id=1], id:1}
size = number: 84
flows = number: 4
(8) global C function 'xpcall'
(9) Lua upvalue 'slave' at file '/home/torre/MoonGen/build/../libmoon/lua/main.lua:128'
Local variables:
args = table: 0x413ae640 {1:2, 2:timerSlave, 3:[TxQueue: id=0, qid=1], 4:[RxQueue: id=1, qid=1], 5:60 (more...)}
ok = boolean: true
taskId = number: 2
func = string: "timerSlave"
(temporary) = table: 0x413b3540 {}
(temporary) = C function: select
(*temporary) = number: 2
(10) Lua function 'main' at file '/home/torre/MoonGen/build/../libmoon/lua/main.lua:148' (best guess)
Local variables:
task = string: "slave"
/home/torre/MoonGen/build/../libmoon/lua/device.lua:477: tx timestamping is not supported by the hardware or driver
[Device: id=0] TX: 3.37 Mpps, 1940 Mbit/s (2479 Mbit/s with framing)
[Device: id=1] RX: 0.00 Mpps, 0 Mbit/s (0 Mbit/s with framing)
[Device: id=0] TX: 3.31 Mpps, 1907 Mbit/s (2436 Mbit/s with framing)
[Device: id=1] RX: 0.00 Mpps, 0 Mbit/s (0 Mbit/s with framing)
[Device: id=0] TX: 3.35 Mpps, 1929 Mbit/s (2465 Mbit/s with framing)
[Device: id=1] RX: 0.00 Mpps, 0 Mbit/s (0 Mbit/s with framing)
^C[Device: id=0] TX: 3.33 (StdDev 0.03) Mpps, 1918 (StdDev 16) Mbit/s (2451 Mbit/s with framing), total 11471042 packets with 825915024 bytes (incl. CRC)
[Device: id=1] RX: 0.00 (StdDev 0.00) Mpps, 0 (StdDev 0) Mbit/s (0 Mbit/s with framing), total 2 packets with 120 bytes (incl. CRC)
Hello! I am trying to use MoonGen to measure the latency of an experimental network. In fact I want to run this packet generator in a VM, as I have read and saw different issues here, I know that do it in virtualized NICs is not the best idea but I really need it.
I have solve some problems related to this, the first is the number of rx queues. If is useful for someone, first you have to enable multiqueue in the Host using "ethtool":
But now I have the problem that the virtual NICs does not support timestamping, I think that is due to de drivers ("virtio"). I don´t Know if there is any solution to this. Also I have though to use the virtual driver e1000e of intel, but with this i can´t set up the multiqueue.
Any solution? This is the output of executing the "l3-load-latency" example:
root@torre:/home/torre/MoonGen# ./build/MoonGen examples/l3-load-latency.lua 0 1[INFO] Initializing DPDK. This will take a few seconds... EAL: Detected 4 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: PCI device 0000:01:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: PCI device 0000:07:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio EAL: using IOMMU type 8 (No-IOMMU) EAL: PCI device 0000:08:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 1af4:1041 net_virtio [INFO] Found 2 usable devices: Device 0: 52:54:00:CF:53:8E (Red Hat, Inc Virtio network device) Device 1: 52:54:00:E2:C7:58 (Red Hat, Inc Virtio network device) [INFO] Waiting for devices to come up... [INFO] Device 1 (52:54:00:E2:C7:58) is up: 10000 MBit/s [INFO] Device 0 (52:54:00:CF:53:8E) is up: 10000 MBit/s [INFO] 2 devices are up. [WARN] Per-queue rate limit is not supported on this device, setting per-device rate limit to 9999 Mbit/s instead (note: this may fail as well if the NIC doesn't support any rate limiting). [WARN] global rate limiting is not supported by the hardware or driver [INFO] Performing ARP lookup on 10.1.0.10 [INFO] Performing ARP lookup on 10.1.0.10 [WARN] l2 filter error: Operation not supported [WARN] l2 filter error: Operation not supported [INFO] Destination mac: 52:54:00:e2:c7:58 [WARN] Packet size 60 is smaller than minimum timestamp size 84. Timestamped packets will be larger than load packets. [INFO] Destination mac: 52:54:00:e2:c7:58 [FATAL] Lua error in task timerSlave /home/torre/MoonGen/build/../libmoon/lua/device.lua:477: tx timestamping is not supported by the hardware or driver Stack Traceback ############### (2) Lua function 'log' at file '/home/torre/MoonGen/build/../libmoon/lua/log.lua:129' (best guess) Local variables: self = table: 0x411c7f98 {DEBUG:0, fatal:function: 0x408c4f70, writeToLog:function: 0x408c4f50, INFO:1 (more...)} str = string: "tx timestamping is not supported by the hardware or driver" (3) Lua method 'unsupported' at file '/home/torre/MoonGen/build/../libmoon/lua/device.lua:477' Local variables: self = [Device: id=0] {initialized:true, rxQueues:table: 0x413b1790, txQueues:table: 0x413b17e0 (more...)} operation = string: "tx timestamping" level = string: "fatal" (4) Lua method 'enableTxTimestamps' at file '/home/torre/MoonGen/build/../libmoon/lua/device.lua:502' Local variables: self = [Device: id=0] {initialized:true, rxQueues:table: 0x413b1790, txQueues:table: 0x413b17e0 (more...)} queue = [TxQueue: id=0, qid=1] {qid:1, dev:[Device: id=0], id:0} (5) Lua method 'enableTimestamps' at file '/home/torre/MoonGen/build/../libmoon/lua/device.lua:577' Local variables: self = [TxQueue: id=0, qid=1] {qid:1, dev:[Device: id=0], id:0} (6) Lua method 'newUdpTimestamper' at file '/home/torre/MoonGen/build/../libmoon/lua/timestamping.lua:34' Local variables: self = table: 0x41fea070 {newUdpTimestamper:function: 0x41fd3ee8, syncClocks:function: 0x41fd6278 (more...)} txQueue = [TxQueue: id=0, qid=1] {qid:1, dev:[Device: id=0], id:0} rxQueue = [RxQueue: id=1, qid=1] {qid:1, dev:[Device: id=1], id:1} udp = boolean: true doNotConfigureUdpPort = nil (7) Lua function 'timerSlave' at file 'examples/l3-load-latency.lua:113' (best guess) Local variables: txQueue = [TxQueue: id=0, qid=1] {qid:1, dev:[Device: id=0], id:0} rxQueue = [RxQueue: id=1, qid=1] {qid:1, dev:[Device: id=1], id:1} size = number: 84 flows = number: 4 (8) global C function 'xpcall' (9) Lua upvalue 'slave' at file '/home/torre/MoonGen/build/../libmoon/lua/main.lua:128' Local variables: args = table: 0x413ae640 {1:2, 2:timerSlave, 3:[TxQueue: id=0, qid=1], 4:[RxQueue: id=1, qid=1], 5:60 (more...)} ok = boolean: true taskId = number: 2 func = string: "timerSlave" (temporary) = table: 0x413b3540 {} (temporary) = C function: select (*temporary) = number: 2 (10) Lua function 'main' at file '/home/torre/MoonGen/build/../libmoon/lua/main.lua:148' (best guess) Local variables: task = string: "slave" /home/torre/MoonGen/build/../libmoon/lua/device.lua:477: tx timestamping is not supported by the hardware or driver [Device: id=0] TX: 3.37 Mpps, 1940 Mbit/s (2479 Mbit/s with framing) [Device: id=1] RX: 0.00 Mpps, 0 Mbit/s (0 Mbit/s with framing) [Device: id=0] TX: 3.31 Mpps, 1907 Mbit/s (2436 Mbit/s with framing) [Device: id=1] RX: 0.00 Mpps, 0 Mbit/s (0 Mbit/s with framing) [Device: id=0] TX: 3.35 Mpps, 1929 Mbit/s (2465 Mbit/s with framing) [Device: id=1] RX: 0.00 Mpps, 0 Mbit/s (0 Mbit/s with framing) ^C[Device: id=0] TX: 3.33 (StdDev 0.03) Mpps, 1918 (StdDev 16) Mbit/s (2451 Mbit/s with framing), total 11471042 packets with 825915024 bytes (incl. CRC) [Device: id=1] RX: 0.00 (StdDev 0.00) Mpps, 0 (StdDev 0) Mbit/s (0 Mbit/s with framing), total 2 packets with 120 bytes (incl. CRC)
Thanks!