search

jpetazzo / pipework

Software-Defined Networking tools for LXC (LinuX Containers)
Apache License 2.0
4.22k stars 727 forks source link

pleth* and veth* interfaces should be ignored by NetworkManager #72

Closed bprodoehl closed 9 years ago

bprodoehl commented 10 years ago

When using pipework to create a lot of virtual interfaces (~100), NetworkManager (on Ubuntu and Fedora) ends up just flipping out over all the new connections. I thought it was supposed to ignore the veth interfaces created by Docker, but it sure seems to be trying to connect on all the veth* and pleth* interfaces. Is there an easy way to have NetworkManager ignore these by wildcard? Is that out of scope for what the pipework script should do?

jpetazzo commented 10 years ago

Whoa that's a great question. I never used NetworkManager and I don't have it installed on any of my machines :/

I hope someone can chime in!

bprodoehl commented 10 years ago

Just as a test, I hacked up pipework to add this line, which effectively makes NetworkManager ignore the pleth interfaces, but occasionally it fails because NetworkManager hasn't found out about the pleth interface at the time this call happens. nmcli is already slow enough, and adding some polling here sounds awful, so I'm back to digging around looking for how to just make NetworkManager ignore by pattern.

diff --git a/pipework b/pipework
index 59e4ecc..c11df30 100755
--- a/pipework
+++ b/pipework
@@ -192,6 +192,7 @@ MTU=$(ip link show $IFNAME | awk '{print $5}')
     case "$BRTYPE" in
         linux)
             (ip link set $LOCAL_IFNAME master $IFNAME > /dev/null 2>&1) || (brc
+            nmcli dev disconnect $LOCAL_IFNAME
             ;;
         openvswitch)
             ovs-vsctl add-port $IFNAME $LOCAL_IFNAME ${VLAN:+"tag=$VLAN"}
jpetazzo commented 10 years ago

OK. Thanks a lot for diving into this; much appreciated!

rade commented 10 years ago

Actually whether or not Network Manager displays an interface seems to be purely down to naming... make the interface name start with 'veth' and it gets ignored. Yes, really. That's why, say, the interfaces created by docker don't show up, since they are all named 'veth'.

So at https://github.com/jpetazzo/pipework/blob/master/pipework#L189, if instead of

LOCAL_IFNAME=pl$NSPID$CONTAINER_IFNAME
GUEST_IFNAME=pg$NSPID$CONTAINER_IFNAME

we write

LOCAL_IFNAME=v${CONTAINER_IFNAME}pl$NSPID
GUEST_IFNAME=v${CONTAINER_IFNAME}pg$NSPID

then, assuming $CONTAINER_IFNAME starts with 'eth', which it usually does, Network Manager remains blissfully unaware.

bprodoehl commented 10 years ago

Thanks for the input. What distro are you seeing that on? At least on Fedora 20 (NetworkManager 0.9.9), I'm seeing this for the output of nmcli dev when I have about 20 containers up. NetworkManager is definitely not ignoring the veth* ones, either, so I'm guessing there's a distro patch to ignore those.

$ nmcli dev
DEVICE      TYPE      STATE                                 
wlp2s0      wifi      connected                             
pl2312eth1  ethernet  connecting (getting IP configuration) 
pl2312eth2  ethernet  connecting (getting IP configuration) 
pl2356eth1  ethernet  connecting (getting IP configuration) 
pl2356eth2  ethernet  connecting (getting IP configuration) 
pl2356eth3  ethernet  connecting (getting IP configuration) 
pl2412eth1  ethernet  connecting (getting IP configuration) 
pl2473eth1  ethernet  connecting (getting IP configuration) 
pl2576eth1  ethernet  connecting (getting IP configuration) 
pl2669eth1  ethernet  connecting (getting IP configuration) 
pl2669eth2  ethernet  connecting (getting IP configuration) 
pl2762eth1  ethernet  connecting (getting IP configuration) 
pl2762eth2  ethernet  connecting (getting IP configuration) 
pl2762eth3  ethernet  connecting (getting IP configuration) 
pl2897eth1  ethernet  connecting (getting IP configuration) 
pl2897eth2  ethernet  connecting (getting IP configuration) 
pl2977eth1  ethernet  connecting (getting IP configuration) 
pl2977eth2  ethernet  connecting (getting IP configuration) 
pl2977eth3  ethernet  connecting (getting IP configuration) 
pl3037eth1  ethernet  connecting (getting IP configuration) 
pl3037eth2  ethernet  connecting (getting IP configuration) 
pl3103eth1  ethernet  connecting (getting IP configuration) 
pl3103eth2  ethernet  connecting (getting IP configuration) 
pl3103eth3  ethernet  connecting (getting IP configuration) 
pl3249eth1  ethernet  connecting (getting IP configuration) 
pl3249eth2  ethernet  connecting (getting IP configuration) 
pl3383eth1  ethernet  connecting (getting IP configuration) 
pl3383eth2  ethernet  connecting (getting IP configuration) 
pl3383eth3  ethernet  connecting (getting IP configuration) 
pl3563eth1  ethernet  connecting (getting IP configuration) 
pl3563eth2  ethernet  connecting (getting IP configuration) 
pl3649eth1  ethernet  connecting (getting IP configuration) 
pl3649eth2  ethernet  connecting (getting IP configuration) 
pl3649eth3  ethernet  connecting (getting IP configuration) 
pl3744eth1  ethernet  connecting (getting IP configuration) 
pl3744eth2  ethernet  connecting (getting IP configuration) 
pl3878eth1  ethernet  connecting (getting IP configuration) 
pl3878eth2  ethernet  connecting (getting IP configuration) 
pl3878eth3  ethernet  connecting (getting IP configuration) 
pl3966eth1  ethernet  connecting (getting IP configuration) 
pl3966eth2  ethernet  connecting (getting IP configuration) 
pl3966eth3  ethernet  connecting (getting IP configuration) 
pl3966eth4  ethernet  connecting (getting IP configuration) 
pl4113eth1  ethernet  connecting (getting IP configuration) 
pl4113eth2  ethernet  connecting (getting IP configuration) 
pl4113eth3  ethernet  connecting (getting IP configuration) 
pl4223eth1  ethernet  connecting (getting IP configuration) 
pl4223eth2  ethernet  connecting (getting IP configuration) 
pl4223eth3  ethernet  connecting (getting IP configuration) 
pl4354eth1  ethernet  connecting (getting IP configuration) 
pl4354eth2  ethernet  connecting (getting IP configuration) 
pl4354eth3  ethernet  connecting (getting IP configuration) 
veth04a1    ethernet  connecting (getting IP configuration) 
veth2d7d    ethernet  connecting (getting IP configuration) 
veth4669    ethernet  connecting (getting IP configuration) 
veth4d7e    ethernet  connecting (getting IP configuration) 
veth4f92    ethernet  connecting (getting IP configuration) 
veth6b04    ethernet  connecting (getting IP configuration) 
veth6bb9    ethernet  connecting (getting IP configuration) 
veth76f0    ethernet  connecting (getting IP configuration) 
veth88ea    ethernet  connecting (getting IP configuration) 
veth8ffd    ethernet  connecting (getting IP configuration) 
veth9e48    ethernet  connecting (getting IP configuration) 
vethbad9    ethernet  connecting (getting IP configuration) 
vethc65f    ethernet  connecting (getting IP configuration) 
vethd054    ethernet  connecting (getting IP configuration) 
vetheeb1    ethernet  connecting (getting IP configuration) 
vethfcc4    ethernet  connecting (getting IP configuration) 
p33p1       ethernet  unavailable                           
lo          loopback  unmanaged

There is some upstream work to ignore them, and this patch would probably do the trick: http://fpaste.org/118551/14055382/

In the meantime, I've been running this script a few seconds after all the pipework commands.

#!/bin/bash
which nmcli
if [ $? -eq 0 ]
then
    nmcli dev | grep "^pl" | cut -d' ' -f 1 | xargs -r -n 1 nmcli dev disconnect
    nmcli dev | grep "^pg" | cut -d' ' -f 1 | xargs -r -n 1 nmcli dev disconnect
    nmcli dev | grep "^veth" | cut -d' ' -f 1 | xargs -r -n 1 nmcli dev disconnect
fi
bprodoehl commented 10 years ago

This is Ubuntu's patch to ignore veth*: http://bazaar.launchpad.net/~network-manager/network-manager/ubuntu/view/head:/debian/patches/add-veth-support.diff

Moving pipework to prefix its interfaces with veth as suggested by @rade would be a good move for most users.

jpetazzo commented 10 years ago

That sounds like a great workaround, thanks.