If the host machine has iptables enabled, then mrun wont guaranty connectivity between Qemu instances and in case iptablesFORWARD Chain is not capable of forwarding the ns3 network packets the network between Qemu instances doesn't work.
How to reproduce
To reproduce this issue it is enough to configure iptables in a way it drops or rejects packets which are assumed to be forwarded and then try to run multiple Qemu instances with mrun configured to be run in an ns3 network.
Proposed Solutions
Add a piece of code in mrun after bringing up the bridges to add ACCEPT forwarding rules to iptables and remove these rules while tearing down the network.
Add an administrative gotcha in documentation or mrun help.
Also there's a workaround to overcome this issue by disabling iptables for Linux bridges. Which can be done like the following:
~# echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
Further Details
I am currently using the commit 4d938fa97c6984ab3d5f7c9b52f890ecfec72593 from qflex and I have already pulled images from install-docker-update branch of the images sub-module.
I am using this command to run the multiple Qemu instances:
~# ./mrun -r qemu-setup-sample-file.xml -qmp -ns /home/aasgari/Documents/qflex/3rdparty/ns3
Also the output from ~# iptables -L -v which contains iptables rules and stats, for the FORWARD chain after sending 10000 UDP packets from one of the Qemu instances to the other just after the operation was done and having it reset before the operation is the following (I had already initialized eth0 for each Qemu instance with local IPs and used echo hello > /dev/udp/[other qemu IP]/[port] to send the UDP packets):
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
10000 340K DOCKER-USER all -- any any anywhere anywhere
10000 340K DOCKER-ISOLATION-STAGE-1 all -- any any anywhere anywhere
0 0 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any docker0 anywhere anywhere
0 0 ACCEPT all -- docker0 !docker0 anywhere anywhere
0 0 ACCEPT all -- docker0 docker0 anywhere anywhere
0 0 ACCEPT all -- any br-e68495d486fc anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any br-e68495d486fc anywhere anywhere
0 0 ACCEPT all -- br-e68495d486fc !br-e68495d486fc anywhere anywhere
0 0 ACCEPT all -- br-e68495d486fc br-e68495d486fc anywhere anywhere
0 0 ACCEPT all -- any br-dc42555c70a5 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any br-dc42555c70a5 anywhere anywhere
0 0 ACCEPT all -- br-dc42555c70a5 !br-dc42555c70a5 anywhere anywhere
0 0 ACCEPT all -- br-dc42555c70a5 br-dc42555c70a5 anywhere anywhere
0 0 ACCEPT all -- any br-993cfd405bb4 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any br-993cfd405bb4 anywhere anywhere
0 0 ACCEPT all -- br-993cfd405bb4 !br-993cfd405bb4 anywhere anywhere
0 0 ACCEPT all -- br-993cfd405bb4 br-993cfd405bb4 anywhere anywhere
0 0 ACCEPT all -- any br-42830f4fa203 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any br-42830f4fa203 anywhere anywhere
0 0 ACCEPT all -- br-42830f4fa203 !br-42830f4fa203 anywhere anywhere
0 0 ACCEPT all -- br-42830f4fa203 br-42830f4fa203 anywhere anywhere
0 0 ACCEPT all -- any virbr0 anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- virbr0 any 192.168.122.0/24 anywhere
0 0 ACCEPT all -- virbr0 virbr0 anywhere anywhere
0 0 REJECT all -- any virbr0 anywhere anywhere reject-with icmp-port-unreachable
0 0 REJECT all -- virbr0 any anywhere anywhere reject-with icmp-port-unreachable
0 0 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- lo any anywhere anywhere
10000 340K FORWARD_direct all -- any any anywhere anywhere
10000 340K FORWARD_IN_ZONES_SOURCE all -- any any anywhere anywhere
10000 340K FORWARD_IN_ZONES all -- any any anywhere anywhere
10000 340K FORWARD_OUT_ZONES_SOURCE all -- any any anywhere anywhere
10000 340K FORWARD_OUT_ZONES all -- any any anywhere anywhere
0 0 DROP all -- any any anywhere anywhere ctstate INVALID
10000 340K REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Which sounds like the last rule is applied and the UDP packets are rejected.
Description
If the host machine has
iptablesenabled, thenmrunwont guaranty connectivity betweenQemuinstances and in caseiptablesFORWARD Chainis not capable of forwarding thens3network packets the network betweenQemuinstances doesn't work.How to reproduce
To reproduce this issue it is enough to configure
iptablesin a way it drops or rejects packets which are assumed to be forwarded and then try to run multipleQemuinstances withmrunconfigured to be run in anns3network.Proposed Solutions
mrunafter bringing up the bridges to add ACCEPT forwarding rules toiptablesand remove these rules while tearing down the network.mrunhelp.Also there's a workaround to overcome this issue by disabling
iptablesfor Linux bridges. Which can be done like the following:~# echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptablesFurther Details
I am currently using the commit
4d938fa97c6984ab3d5f7c9b52f890ecfec72593from qflex and I have already pulled images frominstall-docker-updatebranch of theimagessub-module.I am using this command to run the multiple Qemu instances:
~# ./mrun -r qemu-setup-sample-file.xml -qmp -ns /home/aasgari/Documents/qflex/3rdparty/ns3My
mrunconfiguration files are like these:Also the output from
~# iptables -L -vwhich containsiptablesrules and stats, for theFORWARDchain after sending 10000 UDP packets from one of the Qemu instances to the other just after the operation was done and having it reset before the operation is the following (I had already initializedeth0for each Qemu instance with local IPs and usedecho hello > /dev/udp/[other qemu IP]/[port]to send the UDP packets):Which sounds like the last rule is applied and the UDP packets are rejected.