search

weaveworks / weave

Simple, resilient multi-host containers networking and more.
https://www.weave.works
Apache License 2.0
6.62k stars 669 forks source link

how best to monitor a weave router and its network? #162

Closed hesco closed 9 years ago

hesco commented 9 years ago

Again I am working to resolve an issue manifesting as being unable to ping between IP's on a weave bridge across physical docker hosts. I'm curious to know, particularly from those who built this or have relevant experience using it or who at least are more familiar than I with the internal workings of the weave router:

What is the best way to monitor a weave router and a weave network for issues requiring intervention?

What I learn from this I hope to capture as a sensu check, perhaps even as a sensu metrics script for contribution back to the sensu-community-plugins collection, which as of now has no coverage for a weave network: https://github.com/sensu/sensu-community-plugins

Looking forward to hearing folks ideas.

-- Hugh

bboreham commented 9 years ago

Hi, could I ask for a little clarification about what you are looking for?

Most of the issues I have had personally have been bad configuration - I assigned a duplicate IP address, or I assigned a subnet which wouldn't route to the other side. So not really things you would monitor for - if you got any of those things wrong it would never work.

You could run any kind of pinger inside a container, to detect that connectivity used to work and now no longer works. But it sounds like you are looking for more than that.

hesco commented 9 years ago

I want to know:

is the weave router up and functioning? can is route traffic between containers on a host? can it route traffic between containers across the bridge to other peers? can I collect traffic metrics from the router without having to collect those stats from iptables?

The issue which I was dealing with today had nothing to do with a misconfiguration. On three occasions now I have had to kill and re-daemonize docker to recover from a cross-peer routing issue which manifests as being unable to weave launch after a weave reset. The weave launch would fail complaining of that port 6783 was not available, although netstat -anp | grep :6783 would show no process listening on that port. For further details, see: https://github.com/zettio/weave/issues/158

As I begin to roll out monitoring in my docker cluster (using sensuapp.org), I would like to see not only an alert that one or more sites are throwing 503 errors, but would also like to see alerts that my weave bridge is at the root of the broken-ness so I do not spend time spiraling in on that realization, which might better be spent restoring services before clients start calling me.

Is there anything built into the weave router which might already be doing health checks on its connections with peers, which I could hook into when I build the sensu monitoring checks and metrics collectors?

Does that clarify?

-- Hugh

bboreham commented 9 years ago

Thanks, that's very helpful.

is the weave router up and functioning?

weave status

can is route traffic between containers on a host?

This type of traffic will route within the Linux kernel, over the weave bridge. (Quick note on terminology: I'm using the word 'bridge' here for a specific part of the Linux networking stack, as described here or here. Not the part of weave which transports packets between hosts)

can it route traffic between containers across the bridge to other peers?

Again, weave status will show the connections between peers (routers), and also show those container MAC addresses that have seen traffic in the last 10 minutes. link to documentation

can I collect traffic metrics from the router without having to collect those stats from iptables?

Not at present. It's been discussed. How would you like to collect the metrics - would a http (REST) interface be suitable? Do you use SNMP?

Is there anything built into the weave router which might already be doing health checks on its connections with peers

If the control connection between two peers is broken, it will move from 'Peers' to 'Reconnections' in the weave status output. However that connection is over TCP, and TCP will not drop a connection for a long time if it just stops passing packets.

To me, the best test to see if weave is routing packets is to send some packets over the network, i.e. run something like ping periodically. This will always be better than asking the implementation what it thinks - if the implementation is confused or broken it may lie.

The weave launch would fail complaining of that port 6783 was not available

I see that was discussed at #158; if you get any more data about the circumstances where this happens it may help to track it down.

throwing 503 errors

Sorry, where does that come from?

rade commented 9 years ago

If the control connection between two peers is broken...

Note however that the control connection is only really needed when there are topology changes. Container traffic goes over the UDP "connection". That has heartbeats to keep any NAT paths alive, though, iirc, we don't declare connections as dead on missed heartbeats.

We should conduct an experiments to see what happens when TCP and/or UDP connections between peers go 'stale'. I suspect we may have to add some aliveness checks to the protocol.

How would you like to collect the metrics

And what metrics would you like to collect? In particular, what metrics that aren't readily available from the likes of iptables?

if you get any more data about the circumstances [of weave complaining that 6783 is not avaiable] it may help to track it down

I'd be interested to see what iptable rules are in place at the time of this failure.

hesco commented 9 years ago

I want to track metrics on network usage by client for billing purposes and as a security precaution to watch for abberations. That means tracking traffic container-by-container. Watching the statistics collection on the firewall of the docker host surrounding my haproxy container is not sufficient.

For the most part, I am running my docker containers in unprivileged mode, meaning that I am unable to run iptables on the containers. There are a couple of exceptions to this, and I would love to be able to do more in that regard to lock down inter-container communication, but not at the risk of sharing docker host root with a container. Consequently, the iptables firewall rule sets I am running are happening on the docker hosts themselves.

I have been sorting out puppet manifests to manage the iptables rules for me, flushing what is in place and then replicating what docker and weave would have added, then adding in the other rules which make sense to the containers being run on a particular host. So far, this is all in my in-house code, though I do intend to push what I learn out to the hesco/weave project on the puppet forge.

I'm not sure what the firewall rules have to do with whether the weave router can bind to a port on the local docker host.

Here is the rule set currently running on a docker host we have which hosts our puppet master, a postgresql server, our postfix mail relay, and the components of our sensu monitoring (including the rabbitmq server, which is exposed across our internal network. Its been a couple of days since the last time I was unable to re-launch the weave bridge without a docker daemon restart. But I do not have a chaos monkey deliberately running in my network and so do not know if these rules would survive a weave reset right now or not. These rules are evolving continuously still, and are probably not too far off from where they were on the last instance. 

# Generated by iptables-save v1.4.21 on Fri Oct 31 13:23:07 2014
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [139:36567]
:OUTPUT ACCEPT [51:235316]
-A INPUT -p icmp -m comment --comment "00000 accept all icmp requests" -j ACCEPT
-A INPUT -i lo -m comment --comment "00001 accept all to lo interface" -j ACCEPT
-A INPUT -d 127.0.0.0/8 ! -i lo -p tcp -m comment --comment "00002 reject local traffic not on loopback interface" -j REJECT --reject-with icmp-port-unreachable
-A INPUT -s 80.172.240.206/32 -p tcp -m comment --comment "00010 DROP abusive IPs" -j DROP
-A INPUT -p tcp -m multiport --dports 22 -m comment --comment "00022 accept sshd connections" -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 68.168.146.144/29 -p tcp -m multiport --dports 25 -m comment --comment "00025 accept smtp connections for postfix from public IP range" -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -p udp -m multiport --sports 53 -m comment --comment "00053 accept dns connections" -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -s 68.168.146.144/29 -p tcp -m multiport --dports 5432,5433 -m comment --comment "05432 postgresql server among YMD\'s public IPs" -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 67.187.55.109/32 -p tcp -m multiport --dports 5432,5433 -m comment --comment "05432 postgresql server for home office" -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 10.0.1.0/24 -p tcp -m multiport --dports 5432,5433 -m comment --comment "05432 postgresql server for weave bridge" -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -s 68.168.146.144/29 -p tcp -m multiport --dports 5672 -m comment --comment "05672 rabbitmq server listen to YMD\'s public IPs" -j ACCEPT
-A INPUT -s 10.0.1.0/24 -p tcp -m multiport --dports 5672 -m comment --comment "05672 rabbitmq server listen to weave bridge" -j ACCEPT  
-A INPUT -p tcp -m multiport --dports 8139:8140 -m comment --comment "08140 puppet master " -j ACCEPT
-A INPUT -p udp -m multiport --dports 60001:60010 -m comment --comment "60001 allow mosh connections" -j ACCEPT
-A INPUT -m comment --comment "65533 accept related established rules" -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m comment --comment "65534 LOG iptables denied traffic" -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -m comment --comment "65535 reject all other traffic" -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i docker0 -o docker0 -m comment --comment "00100 accept docker0 to docker0 FORWARD traffic" -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -m comment --comment "00100 accept docker0 traffic to other interfaces on FORWARD chain" -j ACCEPT
-A FORWARD -o docker0 -m comment --comment "00100 accept related, established traffic returning to docker0 bridge in FORWARD chain" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -d 172.17.0.11/32 ! -i docker0 -o docker0 -p tcp -m multiport --dports 6783 -m comment --comment "06783 accept and forward weave routing tcp packets to weave router, when not from docker interface" -j ACCEPT
-A FORWARD -d 172.17.0.11/32 ! -i docker0 -o docker0 -p udp -m multiport --dports 6783 -m comment --comment "06783 accept and forward weave routing udp packets to weave router, when not from docker interface" -j ACCEPT
COMMIT
# Completed on Fri Oct 31 13:23:07 2014
# Generated by iptables-save v1.4.21 on Fri Oct 31 13:23:07 2014
*nat
:PREROUTING ACCEPT [34:2443]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [1:60]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:WEAVE - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -m comment --comment "00100 DOCKER table PREROUTING LOCAL traffic" -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -m comment --comment "00100 DOCKER chain, MASQUERADE docker bridge traffic not bound to docker bridge" -j MASQUERADE
-A POSTROUTING -p tcp -m comment --comment "00101 nat table, POSTROUTING chain jumped to WEAVE chain" -j WEAVE
-A DOCKER -p tcp -m multiport --dports 6783 -m comment --comment "00101 nat table, DOCKER chain, weave routing tcp packets jumped to DNAT chain and weave router" -j DNAT --to-destination 172.17.0.11:6783
-A DOCKER -p udp -m multiport --dports 6783 -m comment --comment "00101 nat table, DOCKER chain, weave routing udp packets jumped to DNAT chain and weave router" -j DNAT --to-destination 172.17.0.11:6783
-A WEAVE ! -s 10.0.1.0/24 -p tcp -m comment --comment "00101 nat table, WEAVE chain, MASQUERADE non-weave bridge packets to weave bridge" -j MASQUERADE
-A WEAVE -s 10.0.1.0/24 -p tcp -m comment --comment "00101 nat table, WEAVE chain, MASQUERADE weave bridge packets bound beyond weave bridge" -j MASQUERADE
COMMIT
# Completed on Fri Oct 31 13:23:07 2014
bboreham commented 9 years ago

You can ask Linux for the stats on the weave virtual ethernet device for a container like this:

$ sudo docker exec 184ad6330960 ip -s link show ethwe
25: ethwe: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether ca:cc:d8:f4:02:00 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    5708       46       0       0       0       0      
    TX: bytes  packets  errors  dropped carrier collsns 
    1362       17       0       0       0       0

This is asking Docker to run the ip command inside the container with ID 184ad6330960; as far as I can see it will always run as root. (Note this depends on having the 'ip' command present inside the container)

You can also run a command from the host (doesn't need to exist inside the container) in the network namespace of the container using our docker-ns script, so for instance:

$ sudo .weave/bin/docker-ns 184 ip -s link show ethwe
25: ethwe: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether ca:cc:d8:f4:02:00 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast   
    5815       47       0       0       0       0      
    TX: bytes  packets  errors  dropped carrier collsns 
    1362       17       0       0       0       0