the VM appears to ignore the ATC and tc settings

[borepstein]

Hello all,

I have ATC set up on a VM under VirtualHost. It serves as a gateway for other VM's hooked up to it via an internal network. It is running CentOS 6.

It's WAN (uplink) NIC is eth0, LAN (downlink) is eth1. I have deliberately tried to significantly slow down the client (internal, NAT'ed) VM's that can communicate to the internal only through the ATCD server/gateway. The settings are seemingly in accordance to what I set them in ATCD. Here is the tc output:

[root@atc-gw ~]# tc qdisc ls dev eth0 qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 361306 qdisc netem 8019: parent 1:2 limit 1000 delay 20.0s loss 50% reorder 20% corrupt 90% gap 1 [root@atc-gw ~]# tc qdisc ls dev eth1 qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 152995 qdisc netem 801a: parent 1:2 limit 1000 delay 10.0s loss 80% reorder 30% corrupt 90% gap 1 [root@atc-gw ~]# [root@atc-gw ~]# tc filter show dev eth0 filter parent 1: protocol ip pref 1 fw filter parent 1: protocol ip pref 1 fw handle 0x2 classid 1:2 police 0x19 rate 1000bit burst 12000b mtu 2Kb action drop overhead 0b ref 1 bind 1

[root@atc-gw ~]# tc filter show dev eth1 filter parent 1: protocol ip pref 1 fw filter parent 1: protocol ip pref 1 fw handle 0x2 classid 1:2 police 0x1a rate 2000bit burst 12000b mtu 2Kb action drop overhead 0b ref 1 bind 1

[root@atc-gw ~]#

Note that the rates are set up at 1 and 2 kbit/s for the interfaces.

While they are setup this way the client (another VirtualBox VM) is still able to download files off the internet at the speeds up to 8 MB/s = 64 Mbit/s!

Yes, I have tested to make sure the traffic does indeed go through the GW - when I shut it down or take down NIC's on it it stops!

The setup on it is quite simple in terms of networking - it is a NAT firewall/gw. Here is the iptables config file on it:

Firewall configuration written by system-config-firewall

Manual customization of this file is not recommended.

*filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

ATCUI

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8000 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -i eth1 -o eth0 -j ACCEPT -A FORWARD -i eth0 -o eth1 -j ACCEPT -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT

*nat :PREROUTING ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] -A POSTROUTING -o eth0 -j MASQUERADE COMMIT

I am truly mystified.

Any help dealing with this will be greatly appreciated.

Cheers,

Boris.

[chantra]

Hi, what is the content of the iptables' mangle table as the device is being shaped? You should see entries for the device's ip being marked. Also dumping the atcd logs somewhere like pastebin could help.they may be in /var/log/daemon.log if not check in other log files.

[borepstein]

Hello cnatra,

Here we go:

[atc@atc-gw ~]$ sudo iptables -t mangle --list -n [sudo] password for atc: Chain PREROUTING (policy ACCEPT) target prot opt source destination

Chain INPUT (policy ACCEPT) target prot opt source destination

Chain FORWARD (policy ACCEPT) target prot opt source destination MARK all -- 0.0.0.0/0 10.5.4.127 MARK set 0x2 MARK all -- 10.5.4.127 0.0.0.0/0 MARK set 0x2

Chain OUTPUT (policy ACCEPT) target prot opt source destination

Chain POSTROUTING (policy ACCEPT) target prot opt source destination [atc@atc-gw ~]$

10.5.4.127 is the machine I am using to access the web interface so I think this makes sense.

Cheers,

Boris.

[borepstein]

Here is the atcd output that far (both in the shell window and in /var/log/messages):

(venv)[atc@atc-gw ~]$ sudo ~/dev/atc/venv/bin/atcd INFO:AtcdVService.AtcdNBServerTask:AtcdNBServerTask Server Started on 127.0.0.1:9090 INFO:AtcdVService.AtcdLinuxShaper:Calling initialize_shaping_system INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -F FORWARD INFO:AtcdVService.AtcdLinuxShaper:deleting root QDisc on eth1 WARNING:AtcdVService.AtcdLinuxShaper:could not delete root QDisc. There might have been nothing to delete INFO:AtcdVService.AtcdLinuxShaper:setting root qdisc on eth1 INFO:AtcdVService.AtcdLinuxShaper:deleting root QDisc on eth0 WARNING:AtcdVService.AtcdLinuxShaper:could not delete root QDisc. There might have been nothing to delete INFO:AtcdVService.AtcdLinuxShaper:setting root qdisc on eth0 INFO:AtcdVService.AtcdLinuxShaper:Restoring shaped connection from DB INFO:AtcdVService.AtcdLinuxShaper:Request startShaping TrafficControl(device=TrafficControlledDevice(controllingIP='10.5.4.127', controlledIP='10.5.4.127'), timeout=19931.442123889923, settings=TrafficControlSetting(down=Shaping(loss=Loss(percentage=80.0, correlation=0.0), delay=Delay(delay=10000, jitter=0, correlation=0.0), rate=2, iptables_options=[], corruption=Corruption(percentage=90.0, correlation=0.0), reorder=Reorder(percentage=30.0, correlation=0.0, gap=0)), up=Shaping(loss=Loss(percentage=50.0, correlation=0.0), delay=Delay(delay=20000, jitter=0, correlation=0.0), rate=1, iptables_options=[], corruption=Corruption(percentage=90.0, correlation=0.0), reorder=Reorder(percentage=20.0, correlation=0.0, gap=0)))) INFO:AtcdVService.AtcdLinuxShaper:Shaping ip 10.5.4.127 on interface eth0 INFO:AtcdVService.AtcdLinuxShaper:create new HTB class on IFID eth0, classid 1:2,parent 1:0, rate 1kbits INFO:AtcdVService.AtcdLinuxShaper:create new Netem qdisc on IFID eth0, parent 1:2, loss 50.0%, delay 20000000 INFO:AtcdVService.AtcdLinuxShaper:create new FW filter on IFID eth0, classid 1:2, handle 2, rate: 1kbits INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -A FORWARD -d 10.5.4.127 -i eth0 -j MARK --set-mark 2 INFO:AtcdVService.AtcdLinuxShaper:Shaping ip 10.5.4.127 on interface eth1 INFO:AtcdVService.AtcdLinuxShaper:create new HTB class on IFID eth1, classid 1:2,parent 1:0, rate 2kbits INFO:AtcdVService.AtcdLinuxShaper:create new Netem qdisc on IFID eth1, parent 1:2, loss 80.0%, delay 10000000 INFO:AtcdVService.AtcdLinuxShaper:create new FW filter on IFID eth1, classid 1:2, handle 2, rate: 2kbits INFO:AtcdVService.AtcdLinuxShaper:Running /sbin/iptables -t mangle -A FORWARD -s 10.5.4.127 -i eth1 -j MARK --set-mark 2 DEBUG:AtcdVService:All tasks started DEBUG:AtcdVService:VService Active. Awaiting graceful shutdown. INFO:AtcdVService.AtcdLinuxShaper:Request getCurrentShaping for ip 10.5.4.127 INFO:AtcdVService.AtcdLinuxShaper:Request getCurrentShaping for ip 10.5.4.127

Nothing looking particularly wrong to me - only no effect on the actual NICs' performance.

Cheers,

Boris.

[chantra]

@borepstein thanks for sending this over. That looks right. Can you also send the client and server's ip route. Also, please join #atcd on freenode, it may be easier to work on this interactively. there is a web chat available at https://webchat.freenode.net/

[borepstein]

@chantra Will do.

I just tried to join the webchat, was kicked from it on the basis of it being invitation-only:

Channel names begin with # (corrected automatically). [13:25] -herbert.freenode.net- *\ Notice -- TS for # changed from 1427909149 to 1164523362 [13:25] == borepstein was kicked from # by ChanServ [Invite only channel]

Do you need to invite me somehow first?

Boris.

On Wed, Apr 1, 2015 at 1:22 PM, chantra notifications@github.com wrote:

@borepstein https://github.com/borepstein thanks for sending this over. That looks right. Can you also send the client and server's ip route. Also, please join #atcd on freenode, it may be easier to work on this interactively. there is a web chat available at https://webchat.freenode.net/

— Reply to this email directly or view it on GitHub https://github.com/facebook/augmented-traffic-control/issues/90#issuecomment-88564027 .

[chantra]

@borepstein no, you do not need to be invited. did oyu enter the right channel name? #atcd

[borepstein]

@chantra

Yes, I believe I did. Will try again.

For now here are the routing tables.

Server/gateway:

[atc@atc-gw ~]$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.120.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 10.1.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 eth1 0.0.0.0 10.1.3.10 0.0.0.0 UG 0 0 0 eth0 [atc@atc-gw ~]$

Client:

[administrator@cent1 ~]$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.120.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0 0.0.0.0 192.168.120.1 0.0.0.0 UG 0 0 0 eth0 [administrator@cent1 ~]$

Thanks again for looking into it.

Cheers,

Boris.

[chantra]

@borepstein here is the issue as much I can see from the output.

Client has IP 192.168.120.X and default route through the ATC gw at 192.168.120.1. The gateway as also a WAN interface 10.1.3.X on which you do natting.

The traffic is being shaped for 10.5.4.127 . Is this your host IP? Are you connecting to the ATC UI from the client? or from the host?

[borepstein]

@chantra

Yes, the client routes through the gateway that also does NAT'ing. The client here is on a completely self-contained network - the rest of the private network does not know it exists.

10.5.4.127 is my desktop from which I control the ATC UI. But it is not communication to this address I use to measure the performance.

Right now the throughput is throttled by the server at 1 Kbit/s on one interface and 2 Kbit/s on the other. Hence 1 KBit/s is all you should see when the client is communicating to the internet. But watch this (just ran it):

[administrator@cent1 ~]$ wget http://cosmos.cites.illinois.edu/pub/centos/6.6/os/x86_64/images/install.img --2015-04-01 09:56:36-- http://cosmos.cites.illinois.edu/pub/centos/6.6/os/x86_64/images/install.img Resolving cosmos.cites.illinois.edu... 192.17.174.4 Connecting to cosmos.cites.illinois.edu|192.17.174.4|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 153079808 (146M) [text/plain] Saving to: “install.img”

100%[======================================>] 153,079,808 4.60M/s in 35s

2015-04-01 09:57:11 (4.13 MB/s) - “install.img” saved [153079808/153079808]

[administrator@cent1 ~]$

Boris.

On Wed, Apr 1, 2015 at 1:47 PM, chantra notifications@github.com wrote:

@borepstein https://github.com/borepstein here is the issue as much I can see from the output.

Client has IP 192.168.120.X and default route through the ATC gw at 192.168.120.1. The gateway as also a WAN interface 10.1.3.X on which you do natting.

The traffic is being shaped for 10.5.4.127 . Is this your host IP? Are you connecting to the ATC UI from the client? or from the host?

— Reply to this email directly or view it on GitHub https://github.com/facebook/augmented-traffic-control/issues/90#issuecomment-88572206 .

[chantra]

@borepstein because you control the UI from your desktop, ATC is setting shaping for this IP (which is not routed through). You need to initiate the connection to the UI from the client.

Alternatively, you may want to check how to use the API: https://github.com/facebook/augmented-traffic-control/tree/master/atc/django-atc-api

[borepstein]

@chantra

Thank you very much, this did the trick!

One thing I don't understand, though - or perhaps it is just me failing to fully grasp the concept - is that it seems like controls are applied to all the traffic on the gateway's interfaces; based on that, shouldn't it be shaped the same for all the traffic, not just traffic to and from a particular client?

Boris.

On Wed, Apr 1, 2015 at 2:03 PM, chantra notifications@github.com wrote:

@borepstein https://github.com/borepstein because you control the UI from your desktop, ATC is setting shaping for this IP (which is not routed through). You need to initiate the connection to the UI from the client.

Alternatively, you may want to check how to use the API: https://github.com/facebook/augmented-traffic-control/tree/master/atc/django-atc-api

— Reply to this email directly or view it on GitHub https://github.com/facebook/augmented-traffic-control/issues/90#issuecomment-88577931 .

[chantra]

@borepstein it shapes the traffic on a per IP basis (see the mangle table mark the packets and those marked packaets are being shaped)

Closing as you are up and running now,.