Meter bandwidth

[rgbkrk]

We should meter bandwidth in and out.

Bandwidth in should be the sum of:

• Bandwidth In sending a job to the API node (which could be very big, especially with the python pickling)
• Bandwidth into the container
• Bandwidth out of the container.

The last two can be monitored via their network interface. I'm not sure what tools are available other than summing on a tcpdump of the interface Docker gives them. The other hard bit is determining what kind of traffic it is. Do we want to regard traffic internal to a data center differently than traffic to the whole web?

[rgbkrk]

vnstat is a tool, but we likely need something in go to help us with this.

# vnstat -l -i bond0.101
Monitoring bond0.101...    (press CTRL-C to stop)

   rx:       12 kbit/s    29 p/s          tx:        0 kbit/s     0 p/s^C

 bond0.101  /  traffic statistics

                           rx         |       tx
--------------------------------------+------------------
  bytes                      434 KiB  |          64 KiB
--------------------------------------+------------------
          max              56 kbit/s  |        4 kbit/s
      average           10.99 kbit/s  |     1.62 kbit/s
          min               4 kbit/s  |        0 kbit/s
--------------------------------------+------------------
  packets                       9022  |             184
--------------------------------------+------------------
          max                155 p/s  |           3 p/s
      average                 28 p/s  |           0 p/s
          min                  7 p/s  |           0 p/s
--------------------------------------+------------------
  time                  5.27 minutes

[rgbkrk]

vnStat is a console-based network traffic monitor. It keeps a log of hourly, daily and monthly network traffic for the selected interface(s). However, it isn't a packet sniffer. The traffic information is analyzed from the proc(5) and sys filesystems depending on availability. That way vnStat can be used even without root permissions on most systems.

Well neat, should have assumed that.

[rgbkrk]

Ah, and ifconfig -a can gives you a summary over an interface that I normally ignore:

veth46b4cbb Link encap:Ethernet  HWaddr 26:24:ed:7e:fa:0b
          inet6 addr: fe80::2424:edff:fe7e:fa0b/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:1296 (1.2 KB)

veth59e82d1 Link encap:Ethernet  HWaddr ca:61:f6:8b:9b:74
          inet6 addr: fe80::c861:f6ff:fe8b:9b74/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:1324831 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1258973 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1061029350 (1.0 GB)  TX bytes:382474515 (382.4 MB)

veth76ed421 Link encap:Ethernet  HWaddr fe:90:10:3e:e1:a9
          inet6 addr: fe80::fc90:10ff:fe3e:e1a9/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:648 (648.0 B)

RX bytes:648 (648.0 B) TX bytes:1296 (1.2 KB)

The container still has to be up though.

[rgbkrk]

Now I wish I knew about how the Linux kernel reports this information so we could collect the bandwidth numbers easily. Won't solve the DC problem though...

[rgbkrk]

Welp, we could use a libpcap wrapper for Go and sum the bytes on the interface. Then if we wanted to expose controls to demarcate IP blocks to ignore we could. Not terrible.

[rgbkrk]

Actually, even better is gopacket. That's actually what packetbeat uses.

[rgbkrk]

Yeah, that wasn't too bad to get started.

[smashwilson]

:+1: Awesome! How should we integrate this?

[rgbkrk]

We could determine the interface and do the IP sniffing on container creation, all from the API node.

There's a big issue with trying to monitor the interface from the API node if we're using Docker swarm though. Kind of making me think we'd need to run our own job on each node.

I also have no idea how many CPU cycles or RAM we'd be wasting on monitoring. For now I could integrate it and we roll with it from there, maybe defaulting the monitoring to off.