search

p-quic / pquic

The PQUIC implementation, a framework that enables QUIC clients and servers to dynamically exchange protocol plugins that extend the protocol on a per-connection basis
https://pquic.org
MIT License
126 stars 20 forks source link

Handling 'interface down' events #21

Open nemethf opened 3 years ago

nemethf commented 3 years ago

It seems mp-quic doesn't handle changes in interface status. Case 1 and 2 (below) are almost identical, but one of the client interfaces is turned down at 2s in Case 2. Additionally, the available link capacity is always higher in Case 2 than Case 1. Case 1 finishes in 11s, but Case 2 throws an error at 14.3s.

For sake of reproducibility, the end of the bug report contains a small modification of minitopo.

Thanks!

case 1
topo ``` topoType:MultiIf leftSubnet:10.0. rightSubnet:10.1. path_c2r_0:40,40,2 path_c2r_1:40,40,2 ```
xp ``` xpType:pquic clientPcap:yes serverPcap:yes snaplenPcap:100 pquicPlugins:~/pquic/plugins/multipath/multipath_rr_cond.plugin pquicSize:5120000 pquic_ifdown:0 ```
logs ``` $ mprun -t topo -x ./xp_norm; tail pquic_client.log 2021-05-06 13:11:33,634 [INFO] __init__: Extract parameters from file topo 2021-05-06 13:11:33,635 [WARNING] load_parameter_file: Got error 'need more than 1 value to unpack' for line ' '; ignore it 2021-05-06 13:11:33,638 [INFO] __init__: {'path_c2r_0': '40,40,2', 'path_c2r_1': '40,40,2', 'rightSubnet': '10.1.', 'topoType': 'MultiIf', 'leftSubnet': '10.0.'} Link type: c2r Link id: 0 Delay: 40.0 Queue Size: 40 Bandwidth: 2.0 Loss: 0.0 Backup: 0 Link type: c2r Link id: 1 Delay: 40.0 Queue Size: 40 Bandwidth: 2.0 Loss: 0.0 Backup: 0 2021-05-06 13:11:33,643 [INFO] __init__: Initializing MultiInterfaceTopo... 2021-05-06 13:11:33,646 [INFO] apply_topo: Using topo Simple multiple interface topology /-sw---bl---sw-\ c r s \-sw---bl---sw-/ \-sw---bl---sw-/ 2021-05-06 13:11:33,649 [INFO] apply_topo_config: Using topo config 2021-05-06 13:11:34,234 [INFO] disable_tso: Disable TSO, GSO and GRO on all interfaces of all nodes 2021-05-06 13:11:34,374 [INFO] configure_interfaces: Configure interfaces using MultiInterfaceConfig... 2021-05-06 13:11:34,375 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_1-eth1 root; tc qdisc del dev bs_c2r_0_1-eth1 ingress 2021-05-06 13:11:34,396 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_1-eth2 root; tc qdisc del dev bs_c2r_0_1-eth2 ingress 2021-05-06 13:11:34,410 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_2-eth1 root; tc qdisc del dev bs_c2r_0_2-eth1 ingress 2021-05-06 13:11:34,423 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_2-eth2 root; tc qdisc del dev bs_c2r_0_2-eth2 ingress 2021-05-06 13:11:34,444 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_1-eth2 root handle 10: netem delay 40.0ms limit 50000 2021-05-06 13:11:34,453 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_2-eth2 root handle 1:0 tbf rate 2.0mbit burst 15000 limit 78000.0 2021-05-06 13:11:34,466 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_2-eth1 root handle 10: netem delay 40.0ms limit 50000 2021-05-06 13:11:34,475 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_1-eth1 root handle 1:0 tbf rate 2.0mbit burst 15000 limit 78000.0 2021-05-06 13:11:34,495 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_1-eth1 root; tc qdisc del dev bs_c2r_1_1-eth1 ingress 2021-05-06 13:11:34,510 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_1-eth2 root; tc qdisc del dev bs_c2r_1_1-eth2 ingress 2021-05-06 13:11:34,525 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_2-eth1 root; tc qdisc del dev bs_c2r_1_2-eth1 ingress 2021-05-06 13:11:34,542 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_2-eth2 root; tc qdisc del dev bs_c2r_1_2-eth2 ingress 2021-05-06 13:11:34,558 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_1-eth2 root handle 10: netem delay 40.0ms limit 50000 2021-05-06 13:11:34,574 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_2-eth2 root handle 1:0 tbf rate 2.0mbit burst 15000 limit 78000.0 2021-05-06 13:11:34,582 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_2-eth1 root handle 10: netem delay 40.0ms limit 50000 2021-05-06 13:11:34,596 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_1-eth1 root handle 1:0 tbf rate 2.0mbit burst 15000 limit 78000.0 2021-05-06 13:11:34,664 [INFO] __init__: Extract parameters from file ./xp_norm 2021-05-06 13:11:34,670 [INFO] __init__: Extract parameters from file ./xp_norm 2021-05-06 13:11:34,683 [INFO] ping: ping -c 5 -I 10.0.0.1 10.1.0.1 >> ping.log 2021-05-06 13:11:38,799 [INFO] ping: ping -c 5 -I 10.0.1.1 10.1.0.1 >> ping.log 2021-05-06 13:11:42,959 [INFO] run_tcpdump: Activating tcpdump, waiting for it to run 2021-05-06 13:11:47,963 [INFO] run_netem_at: No need to change netem 2021-05-06 13:11:47,975 [INFO] get_pquic_server_cmd: ~/pquic/picoquicdemo -P ~/pquic/plugins/multipath/multipath_rr_cond.plugin -c ~/pquic/certs/cert.pem -k ~/pquic/certs/key.pem &> pquic_server.log & 2021-05-06 13:11:49,983 [INFO] get_pquic_client_cmd: ~/pquic/picoquicdemo -P ~/pquic/plugins/multipath/multipath_rr_cond.plugin -4 -G 5120000 10.1.0.1 4443 &> pquic_client.log 2021-05-06 13:12:03,797 [INFO] : cleanup mininet bb3da03cb78ee84e: Receiving packet type: 6 (1rtt protected phi0), S1, bb3da03cb78ee84e: , Seq: 5 (5) bb3da03cb78ee84e: Decrypted 4 bytes bb3da03cb78ee84e: connection_close, Error 0x0, Reason length 0 Processed 33 bytes, state = 18 (disconnected), return 0 5120000 bytes received 11022.279 ms Client exit with code = 0 ```
case 2
topo ``` topoType:MultiIf leftSubnet:10.0. rightSubnet:10.1. path_c2r_0:40,40,4 path_c2r_1:40,40,4 ```
xp ``` xpType:pquic clientPcap:yes serverPcap:yes snaplenPcap:100 pquicPlugins:~/pquic/plugins/multipath/multipath_rr_cond.plugin pquicSize:5120000 pquic_ifdown:2 ```
logs ``` $ mprun -t topo2 -x ./xp_down; tail pquic_client.log 2021-05-06 13:13:41,012 [INFO] __init__: Extract parameters from file topo2 2021-05-06 13:13:41,013 [INFO] __init__: {'path_c2r_0': '40,40,4', 'path_c2r_1': '40,40,4', 'rightSubnet': '10.1.', 'topoType': 'MultiIf', 'leftSubnet': '10.0.'} Link type: c2r Link id: 0 Delay: 40.0 Queue Size: 40 Bandwidth: 4.0 Loss: 0.0 Backup: 0 Link type: c2r Link id: 1 Delay: 40.0 Queue Size: 40 Bandwidth: 4.0 Loss: 0.0 Backup: 0 2021-05-06 13:13:41,017 [INFO] __init__: Initializing MultiInterfaceTopo... 2021-05-06 13:13:41,021 [INFO] apply_topo: Using topo Simple multiple interface topology /-sw---bl---sw-\ c r s \-sw---bl---sw-/ \-sw---bl---sw-/ 2021-05-06 13:13:41,021 [INFO] apply_topo_config: Using topo config 2021-05-06 13:13:41,616 [INFO] disable_tso: Disable TSO, GSO and GRO on all interfaces of all nodes 2021-05-06 13:13:41,758 [INFO] configure_interfaces: Configure interfaces using MultiInterfaceConfig... 2021-05-06 13:13:41,759 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_1-eth1 root; tc qdisc del dev bs_c2r_0_1-eth1 ingress 2021-05-06 13:13:41,777 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_1-eth2 root; tc qdisc del dev bs_c2r_0_1-eth2 ingress 2021-05-06 13:13:41,793 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_2-eth1 root; tc qdisc del dev bs_c2r_0_2-eth1 ingress 2021-05-06 13:13:41,814 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_2-eth2 root; tc qdisc del dev bs_c2r_0_2-eth2 ingress 2021-05-06 13:13:41,831 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_1-eth2 root handle 10: netem delay 40.0ms limit 50000 2021-05-06 13:13:41,849 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_2-eth2 root handle 1:0 tbf rate 4.0mbit burst 15000 limit 97500.0 2021-05-06 13:13:41,857 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_2-eth1 root handle 10: netem delay 40.0ms limit 50000 2021-05-06 13:13:41,870 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_1-eth1 root handle 1:0 tbf rate 4.0mbit burst 15000 limit 97500.0 2021-05-06 13:13:41,886 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_1-eth1 root; tc qdisc del dev bs_c2r_1_1-eth1 ingress 2021-05-06 13:13:41,932 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_1-eth2 root; tc qdisc del dev bs_c2r_1_1-eth2 ingress 2021-05-06 13:13:41,945 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_2-eth1 root; tc qdisc del dev bs_c2r_1_2-eth1 ingress 2021-05-06 13:13:41,968 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_2-eth2 root; tc qdisc del dev bs_c2r_1_2-eth2 ingress 2021-05-06 13:13:41,986 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_1-eth2 root handle 10: netem delay 40.0ms limit 50000 2021-05-06 13:13:41,996 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_2-eth2 root handle 1:0 tbf rate 4.0mbit burst 15000 limit 97500.0 2021-05-06 13:13:42,015 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_2-eth1 root handle 10: netem delay 40.0ms limit 50000 2021-05-06 13:13:42,030 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_1-eth1 root handle 1:0 tbf rate 4.0mbit burst 15000 limit 97500.0 2021-05-06 13:13:42,102 [INFO] __init__: Extract parameters from file ./xp_down 2021-05-06 13:13:42,104 [INFO] __init__: Extract parameters from file ./xp_down 2021-05-06 13:13:42,119 [INFO] ping: ping -c 5 -I 10.0.0.1 10.1.0.1 >> ping.log 2021-05-06 13:13:46,232 [INFO] ping: ping -c 5 -I 10.0.1.1 10.1.0.1 >> ping.log 2021-05-06 13:13:50,383 [INFO] run_tcpdump: Activating tcpdump, waiting for it to run 2021-05-06 13:13:55,387 [INFO] run_netem_at: No need to change netem 2021-05-06 13:13:55,399 [INFO] get_pquic_server_cmd: ~/pquic/picoquicdemo -P ~/pquic/plugins/multipath/multipath_rr_cond.plugin -c ~/pquic/certs/cert.pem -k ~/pquic/certs/key.pem &> pquic_server.log & 2021-05-06 13:13:57,409 [INFO] get_pquic_client_cmd: ((sleep 2; ip link set dev Client_0-eth1 down)&);~/pquic/picoquicdemo -P ~/pquic/plugins/multipath/multipath_rr_cond.plugin -4 -G 5120000 10.1.0.1 4443 &> pquic_client.log 2021-05-06 13:14:14,587 [INFO] : cleanup mininet 6e586e5ba058d0cc: Receiving 33 bytes from 10.1.0.1:4443 at T=14.314553 (5c1a916532b20) 6e586e5ba058d0cc: Receiving packet type: 6 (1rtt protected phi0), S1, 6e586e5ba058d0cc: <2533849e9bd76492>, Seq: 8 (8) 6e586e5ba058d0cc: Decrypted 4 bytes 6e586e5ba058d0cc: connection_close, Error 0x0, Reason length 0 Processed 33 bytes, state = 18 (disconnected), return 0 -1.0 Client exit with code = 0 ```
modifications of minitopo ```diff diff --git a/experiments/pquic.py b/experiments/pquic.py index 3ebe491..53acee0 100644 --- a/experiments/pquic.py +++ b/experiments/pquic.py @@ -8,6 +8,7 @@ class PQUICParameter(ExperimentParameter): CLIENT_PLUGINS = "pquicClientPlugins" SERVER_PLUGINS = "pquicServerPlugins" SIZE = "pquicSize" + IF_DOWN = "pquic_ifdown" def __init__(self, experiment_parameter_filename): super(PQUICParameter, self).__init__(experiment_parameter_filename) @@ -16,6 +17,7 @@ class PQUICParameter(ExperimentParameter): PQUICParameter.CLIENT_PLUGINS: "", PQUICParameter.SERVER_PLUGINS: "", PQUICParameter.SIZE: 10240000, + PQUICParameter.IF_DOWN: 0, }) @@ -40,6 +42,7 @@ class PQUIC(Experiment): self.client_plugins = self.experiment_parameter.get(PQUICParameter.CLIENT_PLUGINS) self.server_plugins = self.experiment_parameter.get(PQUICParameter.SERVER_PLUGINS) self.size = int(self.experiment_parameter.get(PQUICParameter.SIZE)) + self.if_down = int(self.experiment_parameter.get(PQUICParameter.IF_DOWN)) def prepare(self): super(PQUIC, self).prepare() @@ -62,7 +65,10 @@ class PQUIC(Experiment): return s def get_pquic_client_cmd(self): - s = "{} {} -4 -G {} {} 4443 &> {}".format(PQUIC.BIN, self.get_plugin_cmd(client=True), self.size, + s = "" + if self.if_down: + s += "((sleep {}; ip link set dev Client_0-eth1 down)&);".format(self.if_down) + s += "{} {} -4 -G {} {} 4443 &> {}".format(PQUIC.BIN, self.get_plugin_cmd(client=True), self.size, self.topo_config.get_server_ip(), PQUIC.CLIENT_LOG) logging.info(s) return s ```
qdeconinck commented 3 years ago

Indeed, the current version of the multipath plugin does not handle address mobility and may stuck with a lost path. This is something we may want to fix someday.