Closed j-breyer closed 2 years ago
Could you please include the full command you're running flent with, and the full log output? I'm not quite sure I understand what you're trying to do and what goes wrong from your description...
I'm running the flent tests in a mininet with IPv6 enabled (ping is working).
On server side:
sudo netserver -6 -L fe80::200:aff:fe00:e02%h2-eth0 -p 12865 -d
to start the netserver, then
sudo irtt server
for the irtt server
On client side:
sudo flent rrul -6 -p all_scaled --local-bind=fe80::200:aff:fe00:601%h1-eth0 -l 60 -H fe80::200:aff:fe00:e02 -t ipv6 -o ipv6_dscp.pdf -v
fails to achieve the UDP RTT measurement, but completes the TCP and ICMP6 part (full verbose output below).
Running the same command with IPv4 addresses of the hosts (netserver enabled respectively):
sudo flent rrul -p all_scaled --local-bind=10.0.6.1 -l 60 -H 10.0.14.2 -t ipv4 -o ipv4_dscp.pdf -v
works just fine.
The difference in output comes in these lines:
Ping (ms) UDP EF: Adding child IrttRunner
which: /usr/local/sbin/irtt is not an executable file
which: /usr/local/bin/irtt is not an executable file
which: /usr/sbin/irtt is not an executable file
which: Found irtt executable at /usr/bin/irtt
UDP RTT test: Cannot use irtt runner (Irtt connection check failed: Error: address fe80::200:aff:fe00:601%h1-eth0:0: too many colons in address
). Using netperf UDP_RR
which comes from flent
passing the IPv6 address not in the correct format to irtt
- irtt
requires the IPv6 passed in [a::b]
brackets, to distinguish it from the port (if I read it correctly, reference to the irtt page
Full output of the IPv6-command:
Executing test environment file /usr/share/flent/flent/tests/rrul.conf
Started Flent 1.3.2 using Python 3.8.10.
Looking up hostname 'fe80::200:aff:fe00:e02'.
Executing test environment file /usr/share/flent/flent/tests/rrul.conf
Gathering local metadata
Looking up hostname 'fe80::200:aff:fe00:e02'.
Starting rrul test. Expected run time: 70 seconds.
which: /usr/local/sbin/netperf is not an executable file
which: /usr/local/bin/netperf is not an executable file
which: /usr/sbin/netperf is not an executable file
which: Found netperf executable at /usr/bin/netperf
Ping (ms) UDP EF: Adding child IrttRunner
which: /usr/local/sbin/irtt is not an executable file
which: /usr/local/bin/irtt is not an executable file
which: /usr/sbin/irtt is not an executable file
which: Found irtt executable at /usr/bin/irtt
UDP RTT test: Cannot use irtt runner (Irtt connection check failed: Error: address fe80::200:aff:fe00:601%h1-eth0:0: too many colons in address
). Using netperf UDP_RR
Ping (ms) UDP EF: Adding child NetperfDemoRunner
Ping (ms) UDP BK: Adding child IrttRunner
which: /usr/local/sbin/irtt is not an executable file
which: /usr/local/bin/irtt is not an executable file
which: /usr/sbin/irtt is not an executable file
which: Found irtt executable at /usr/bin/irtt
UDP RTT test: Cannot use irtt runner (Irtt connection check failed: Error: address fe80::200:aff:fe00:601%h1-eth0:0: too many colons in address
). Using netperf UDP_RR
Ping (ms) UDP BK: Adding child NetperfDemoRunner
Ping (ms) UDP BE: Adding child IrttRunner
which: /usr/local/sbin/irtt is not an executable file
which: /usr/local/bin/irtt is not an executable file
which: /usr/sbin/irtt is not an executable file
which: Found irtt executable at /usr/bin/irtt
UDP RTT test: Cannot use irtt runner (Irtt connection check failed: Error: address fe80::200:aff:fe00:601%h1-eth0:0: too many colons in address
). Using netperf UDP_RR
Ping (ms) UDP BE: Adding child NetperfDemoRunner
which: /usr/local/sbin/fping6 is not an executable file
which: /usr/local/bin/fping6 is not an executable file
which: /usr/sbin/fping6 is not an executable file
which: Found fping6 executable at /usr/bin/fping6
which: /usr/local/sbin/ping6 is not an executable file
which: /usr/local/bin/ping6 is not an executable file
which: /usr/sbin/ping6 is not an executable file
which: Found ping6 executable at /usr/bin/ping6
Forked /usr/bin/netperf as pid 288938
TCP upload BE: Starting watchdog with timeout 75
Started TimerRunner idx 0 ('Watchdog [TCP upload BE]')
Started NetperfDemoRunner idx 0 ('TCP upload BE')
Forked /usr/bin/netperf as pid 288941
TCP upload BK: Starting watchdog with timeout 75
Started TimerRunner idx 1 ('Watchdog [TCP upload BK]')
Started NetperfDemoRunner idx 1 ('TCP upload BK')
Forked /usr/bin/netperf as pid 288944
TCP upload CS5: Starting watchdog with timeout 75
Started TimerRunner idx 2 ('Watchdog [TCP upload CS5]')
Started NetperfDemoRunner idx 2 ('TCP upload CS5')
Forked /usr/bin/netperf as pid 288947
TCP upload EF: Starting watchdog with timeout 75
Started TimerRunner idx 3 ('Watchdog [TCP upload EF]')
Started NetperfDemoRunner idx 3 ('TCP upload EF')
Forked /usr/bin/netperf as pid 288950
TCP download BE: Starting watchdog with timeout 75
Started TimerRunner idx 6 ('Watchdog [TCP download BE]')
Started NetperfDemoRunner idx 6 ('TCP download BE')
Forked /usr/bin/netperf as pid 288953
TCP download BK: Starting watchdog with timeout 75
Started TimerRunner idx 7 ('Watchdog [TCP download BK]')
Started NetperfDemoRunner idx 7 ('TCP download BK')
Forked /usr/bin/netperf as pid 288956
TCP download CS5: Starting watchdog with timeout 75
Started TimerRunner idx 8 ('Watchdog [TCP download CS5]')
Started NetperfDemoRunner idx 8 ('TCP download CS5')
Forked /usr/bin/netperf as pid 288959
TCP download EF: Starting watchdog with timeout 75
Started TimerRunner idx 9 ('Watchdog [TCP download EF]')
Started NetperfDemoRunner idx 9 ('TCP download EF')
Forked /usr/bin/netperf as pid 288962
Ping (ms) UDP EF :: child 0: Starting watchdog with timeout 80
Started TimerRunner idx 13 ('Watchdog [Ping (ms) UDP EF :: child 0]')
Started NetperfDemoRunner idx 13 ('Ping (ms) UDP EF :: child 0')
Forked /usr/bin/netperf as pid 288965
Ping (ms) UDP BK :: child 0: Starting watchdog with timeout 80
Started TimerRunner idx 14 ('Watchdog [Ping (ms) UDP BK :: child 0]')
Started NetperfDemoRunner idx 14 ('Ping (ms) UDP BK :: child 0')
Forked /usr/bin/netperf as pid 288968
Ping (ms) UDP BE :: child 0: Starting watchdog with timeout 80
Started TimerRunner idx 15 ('Watchdog [Ping (ms) UDP BE :: child 0]')
Started NetperfDemoRunner idx 15 ('Ping (ms) UDP BE :: child 0')
Forked /usr/bin/fping6 as pid 288971
Ping (ms) ICMP: Starting watchdog with timeout 71
Started TimerRunner idx 16 ('Watchdog [Ping (ms) ICMP]')
Started PingRunner idx 16 ('Ping (ms) ICMP')
WARNING: Program exited non-zero.
Runner class: NetperfDemoRunner
Command: /usr/bin/netperf -P 0 -v 0 -D -0.20 -6 -Y EF,EF -H fe80::200:aff:fe00:e02 -p 12865 -t UDP_RR -l 70 -F /dev/urandom -L fe80::200:aff:fe00:601%h1-eth0 -- -L fe80::200:aff:fe00:601%h1-eth0 -H fe80::200:aff:fe00:e02 -k THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD
Return code: 1
Stdout: send_data: data send error: errno 22
Stderr: netperf: send_omni: send_data failed: Invalid argument
WARNING: Command produced no valid data.
Runner class: NetperfDemoRunner
Command: /usr/bin/netperf -P 0 -v 0 -D -0.20 -6 -Y EF,EF -H fe80::200:aff:fe00:e02 -p 12865 -t UDP_RR -l 70 -F /dev/urandom -L fe80::200:aff:fe00:601%h1-eth0 -- -L fe80::200:aff:fe00:601%h1-eth0 -H fe80::200:aff:fe00:e02 -k THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD
Return code: 1
Stdout: send_data: data send error: errno 22
Stderr: netperf: send_omni: send_data failed: Invalid argument
NetperfDemoRunner Ping (ms) UDP EF :: child 0 finished
WARNING: Program exited non-zero.
Runner class: NetperfDemoRunner
Command: /usr/bin/netperf -P 0 -v 0 -D -0.20 -6 -Y CS1,CS1 -H fe80::200:aff:fe00:e02 -p 12865 -t UDP_RR -l 70 -F /dev/urandom -L fe80::200:aff:fe00:601%h1-eth0 -- -L fe80::200:aff:fe00:601%h1-eth0 -H fe80::200:aff:fe00:e02 -k THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD
Return code: 1
Stdout: send_data: data send error: errno 22
Stderr: netperf: send_omni: send_data failed: Invalid argument
WARNING: Command produced no valid data.
Runner class: NetperfDemoRunner
Command: /usr/bin/netperf -P 0 -v 0 -D -0.20 -6 -Y CS1,CS1 -H fe80::200:aff:fe00:e02 -p 12865 -t UDP_RR -l 70 -F /dev/urandom -L fe80::200:aff:fe00:601%h1-eth0 -- -L fe80::200:aff:fe00:601%h1-eth0 -H fe80::200:aff:fe00:e02 -k THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD
Return code: 1
Stdout: send_data: data send error: errno 22
Stderr: netperf: send_omni: send_data failed: Invalid argument
NetperfDemoRunner Ping (ms) UDP BK :: child 0 finished
WARNING: Program exited non-zero.
Runner class: NetperfDemoRunner
Command: /usr/bin/netperf -P 0 -v 0 -D -0.20 -6 -H fe80::200:aff:fe00:e02 -p 12865 -t UDP_RR -l 70 -F /dev/urandom -L fe80::200:aff:fe00:601%h1-eth0 -- -L fe80::200:aff:fe00:601%h1-eth0 -H fe80::200:aff:fe00:e02 -k THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD
Return code: 1
Stdout: send_data: data send error: errno 22
Stderr: netperf: send_omni: send_data failed: Invalid argument
WARNING: Command produced no valid data.
Runner class: NetperfDemoRunner
Command: /usr/bin/netperf -P 0 -v 0 -D -0.20 -6 -H fe80::200:aff:fe00:e02 -p 12865 -t UDP_RR -l 70 -F /dev/urandom -L fe80::200:aff:fe00:601%h1-eth0 -- -L fe80::200:aff:fe00:601%h1-eth0 -H fe80::200:aff:fe00:e02 -k THROUGHPUT,LOCAL_CONG_CONTROL,REMOTE_CONG_CONTROL,TRANSPORT_MSS,LOCAL_TRANSPORT_RETRANS,REMOTE_TRANSPORT_RETRANS,LOCAL_SOCKET_TOS,REMOTE_SOCKET_TOS,DIRECTION,ELAPSED_TIME,PROTOCOL,LOCAL_SEND_SIZE,LOCAL_RECV_SIZE,REMOTE_SEND_SIZE,REMOTE_RECV_SIZE,LOCAL_BYTES_SENT,LOCAL_BYTES_RECVD,REMOTE_BYTES_SENT,REMOTE_BYTES_RECVD
Return code: 1
Stdout: send_data: data send error: errno 22
Stderr: netperf: send_omni: send_data failed: Invalid argument
NetperfDemoRunner Ping (ms) UDP BE :: child 0 finished
TimerRunner Watchdog [Ping (ms) ICMP]: timer expired
TimerRunner Watchdog [Ping (ms) ICMP] finished
PingRunner Ping (ms) ICMP killed by kill event
PingRunner Ping (ms) ICMP finished
TimerRunner Watchdog [TCP upload BE]: timer expired
TimerRunner Watchdog [TCP upload BE] finished
NetperfDemoRunner TCP upload BE killed by kill event
TimerRunner Watchdog [TCP upload BK]: timer expired
TimerRunner Watchdog [TCP upload BK] finished
NetperfDemoRunner TCP upload BK killed by kill event
TimerRunner Watchdog [TCP upload CS5]: timer expired
TimerRunner Watchdog [TCP upload CS5] finished
NetperfDemoRunner TCP upload CS5 killed by kill event
TimerRunner Watchdog [TCP upload EF]: timer expired
TimerRunner Watchdog [TCP upload EF] finished
NetperfDemoRunner TCP upload EF killed by kill event
TimerRunner Watchdog [TCP download BE]: timer expired
TimerRunner Watchdog [TCP download BE] finished
NetperfDemoRunner TCP download BE killed by kill event
TimerRunner Watchdog [TCP download BK]: timer expired
TimerRunner Watchdog [TCP download BK] finished
NetperfDemoRunner TCP download BK killed by kill event
TimerRunner Watchdog [TCP download CS5]: timer expired
TimerRunner Watchdog [TCP download CS5] finished
NetperfDemoRunner TCP download CS5 killed by kill event
TimerRunner Watchdog [TCP download EF]: timer expired
TimerRunner Watchdog [TCP download EF] finished
NetperfDemoRunner TCP download EF killed by kill event
NetperfDemoRunner TCP upload BE finished
NetperfDemoRunner TCP upload BK finished
NetperfDemoRunner TCP upload CS5 finished
NetperfDemoRunner TCP upload EF finished
NetperfDemoRunner TCP download BE finished
NetperfDemoRunner TCP download BK finished
NetperfDemoRunner TCP download CS5 finished
NetperfDemoRunner TCP download EF finished
Runner aggregation finished
Data file written to ./rrul-2021-09-27T210930.953628.ipv6.flent.gz.
Creating new PlotFormatter
CACHEDIR=/home/justus/.cache/matplotlib
Loaded backend pdf version unknown.
Initialised matplotlib v3.4.2 on numpy v1.21.1.
No raw data found for series TCP download avg, falling back to computed values
No raw data found for series TCP upload avg, falling back to computed values
No raw data found for series Ping (ms) UDP EF, falling back to computed values
No raw data found for series Ping (ms) UDP BK, falling back to computed values
No raw data found for series Ping (ms) UDP BE, falling back to computed values
No raw data found for series Ping (ms) avg, falling back to computed values
Saving plot to ipv6_dscp.pdf
Assigning font /b'F1' = '/home/j/.local/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf'
Embedding font /home/j/.local/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf.
Writing TrueType font.
Assigning font /b'F1' = '/home/j/.local/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf'
Embedding font /home/j/.local/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf.
Writing TrueType font.
TimerRunner Watchdog [Ping (ms) UDP EF :: child 0]: timer expired
TimerRunner Watchdog [Ping (ms) UDP EF :: child 0] finished
TimerRunner Watchdog [Ping (ms) UDP BK :: child 0]: timer expired
TimerRunner Watchdog [Ping (ms) UDP BK :: child 0] finished
TimerRunner Watchdog [Ping (ms) UDP BE :: child 0]: timer expired
TimerRunner Watchdog [Ping (ms) UDP BE :: child 0] finished
Hi, I guess that creates a bit of a hassle on the flent side as it needs to know the address type in order to surround it with brackets. It comes from Go's requirement for brackets around IPv6 address literals in net.Dial(). I added an issue for irtt, but I don't know if/when that will ever get updated in a Debian package as I'm not maintaining that.
Perhaps a workaround would be to detect IPv6 addresses using Python's ipaddress module and surround it with brackets if necessary before passing it to irtt? I could add that to flent at some point if so...
Perhaps a workaround would be to detect IPv6 addresses using Python's ipaddress module and surround it with brackets if necessary before passing it to irtt? I could add that to flent at some point if so...
That'd be very helpful for the moment!
I am also having troubles with other tests, e.g., rrul46compete
, because it seems sometimes a device interface must be appended to the IPv6 for the apps to work, and sometimes it doesn't, as I'm working with link-local address spaces (i.e., fe80::b%dev1
instead of fe80::b
).
That may be related to the irtt issue, but maybe it's a whole new issue of its own. (If there is no quick fix for that, I could try to use other IPv6 spaces, but that doesn't help with the issue itself)
The proposed fix is in pull request #239.
I tested an rrul test briefly and successfully passed in a link local IPv6 address containing a scope (%dev), so that should work at least as far as irtt is concerned.
I found another workaround by setting up non-scoped IPv6 addresses on the hosts (e.g., a::601
and a::e02
).
Then I added them to /etc/hosts
with a related hostname, e.g., a::601 mn.h1
and a::e02 mn.h2
Running now sudo flent rrul -p all_scaled -6 --local-bind mn.h1 -l 60 -H mn.h2 -o rrul_hosts6.pdf -v
gives Error free output.
However, I noticed a different bug when using this:
If a host contains an IPv4 as well as an IPv6 address in /etc/hosts
, flent seems to always prefer the IPv4 entry. This results in the process either failing, because it tries an irtt connection from IPv4 to IPv6 address, or the above test runs the irtt connection completely in IPv4, ignoring the -6
argument, if both devices can be resolved to an IPv4
Maybe this is content for a separate Issue?
j-breyer @.***> writes:
I found another workaround by setting up non-scoped IPv6 addresses on the hosts (e.g.,
a::601
anda::e02
). Then I added them to/etc/hosts
with a related hostname, e.g.,a::601 mn.h1
anda::e02 mn.h2
Running now
sudo flent rrul -p all_scaled -6 --local-bind mn.h1 -l 60 -H mn.h2 -o rrul_hosts6.pdf -v
gives Error free output.However, I noticed a different bug when using this: If a host contains an IPv4 as well as an IPv6 address in
/etc/hosts
, flent seems to always prefer the IPv4 entry. This results in the process either failing, because it tries an irtt connection from IPv4 to IPv6 address, or the above test runs the irtt connection completely in IPv4, ignoring the-6
argument, if both devices can be resolved to an IPv4Maybe this is content for a separate Issue?
Yes, please open a separate issue. Flent uses the host name resolution mechanism, so in the absence of -4/-6, it should prefer whatever the host does. However, a -6 should override this, so if it doesn't that's a bug...
Also, note that as a general point, a fe80:: address is not well-defined without the interface (as link-local addresses are always link-scoped with the same subnet for IPv6). So anytime you use such an address, you should be including the %devname bit...
Yes, please open a separate issue. Flent uses the host name resolution mechanism, so in the absence of -4/-6, it should prefer whatever the host does. However, a -6 should override this, so if it doesn't that's a bug...
Bug persists with updated version of flent, reported in Issue #241
Thank you for the effort!
Testing flent with IPv6 Addresses yields a similar error to Issue #214, despite installing the irtt tool. (IPv4 works fine with irtt, already tested.)
Running in
-v
to verbose showed me, that the format of IPv6 addresses is false forirtt
: Part of the output:which: Found irtt executable at /usr/bin/irtt
UDP RTT test: Cannot use irtt runner (Irtt connection check failed: Error: address fe80::200:aff:fe00:601%h1-eth0:0: too many colons in address
). Using netperf UDP_RR
Ping (ms) UDP EF: Adding child NetperfDemoRunner
flent is run with
--local-bind
and-H
in my case, for address handover of client and server.Scrolling a bit through the irtt docs, I found that the host format for IPv6 addresses should be in brackets [ ]
Handing over the
--local-bind
in bracket format as a quick test solves the irtt issue with that address, but of course results in the address resolution to failAny help or quick fix is much appreciated!