search

lxc / lxc

LXC - Linux Containers
https://linuxcontainers.org/lxc
Other
4.62k stars 1.12k forks source link

The virtual port in OVS is not deleted after container shutdown #2849

Closed Milan-Benes closed 5 years ago

Milan-Benes commented 5 years ago

Required information

--- Namespaces --- Namespaces: enabled Utsname namespace: enabled Ipc namespace: enabled Pid namespace: enabled User namespace: enabled Network namespace: enabled

--- Control groups --- Cgroups: enabled

Cgroup v1 mount points: /sys/fs/cgroup/systemd /sys/fs/cgroup/blkio /sys/fs/cgroup/perf_event /sys/fs/cgroup/pids /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/hugetlb /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/memory /sys/fs/cgroup/cpuset /sys/fs/cgroup/freezer /sys/fs/cgroup/devices

Cgroup v2 mount points: /sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled Cgroup device: enabled Cgroup sched: enabled Cgroup cpu account: enabled Cgroup memory controller: enabled Cgroup cpuset: enabled

--- Misc --- Veth pair device: enabled, not loaded Macvlan: enabled, not loaded Vlan: enabled, not loaded Bridges: enabled, not loaded Advanced netfilter: enabled, not loaded CONFIG_NF_NAT_IPV4: enabled, not loaded CONFIG_NF_NAT_IPV6: enabled, not loaded CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore --- checkpoint restore: enabled CONFIG_FHANDLE: enabled CONFIG_EVENTFD: enabled CONFIG_EPOLL: enabled CONFIG_UNIX_DIAG: enabled CONFIG_INET_DIAG: enabled CONFIG_PACKET_DIAG: enabled CONFIG_NETLINK_DIAG: enabled File capabilities:

Note : Before booting a new kernel, you can check its configuration usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

data1/ROOT / zfs rw,noatime,xattr,noacl 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,nosuid,size=12196640k,nr_inodes=3049160,mode=755 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 debugfs /sys/kernel/debug debugfs rw,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1312 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /var/tmp/portage tmpfs rw,relatime 0 0 data1 /volumes/data1 zfs rw,noatime,xattr,noacl 0 0 data1/lxc-data1 /volumes/data1/lxc-data1 zfs rw,noatime,xattr,noacl 0 0 data1/lxc-data1/rootfs-data1 /volumes/data1/lxc-data1/rootfs-data1 zfs rw,noatime,xattr,noacl 0 0 data1/lxc-data1/rootfs-data1/bionic-amd64-template /volumes/data1/lxc-data1/rootfs-data1/bionic-amd64-template zfs rw,noatime,xattr,noacl 0 0 data1/lxc-data1/rootfs-data1/centos-7-amd64-template /volumes/data1/lxc-data1/rootfs-data1/centos-7-amd64-template zfs rw,noatime,xattr,noacl 0 0 data1/lxc-data1/rootfs-data1/confluence /volumes/data1/lxc-data1/rootfs-data1/confluence zfs rw,noatime,xattr,noacl 0 0 data1/lxc-data1/rootfs-data1/sonarqube /volumes/data1/lxc-data1/rootfs-data1/sonarqube zfs rw,noatime,xattr,noacl 0 0 data1/lxc-data1/rootfs-data1/teamcity /volumes/data1/lxc-data1/rootfs-data1/teamcity zfs rw,noatime,xattr,noacl 0 0 data1/lxc-data1/rootfs-data1/trusty-amd64-template /volumes/data1/lxc-data1/rootfs-data1/trusty-amd64-template zfs rw,noatime,xattr,noacl 0 0 data1/nr-data /volumes/data1/nr-data zfs rw,noatime,xattr,noacl 0 0 data1/nr-data/portage /volumes/data1/nr-data/portage zfs rw,noatime,xattr,noacl 0 0 data1/nr-data/portage/distfiles /volumes/data1/nr-data/portage/distfiles zfs rw,noatime,xattr,noacl 0 0 data1/nr-data/tmp /volumes/data1/nr-data/tmp zfs rw,nosuid,nodev,noexec,xattr,noacl 0 0 data1/nr-data/usr_src /volumes/data1/nr-data/usr_src zfs rw,noatime,xattr,noacl 0 0 data1/r-data /volumes/data1/r-data zfs rw,noatime,xattr,noacl 0 0 data1/r-data/Dev /volumes/data1/r-data/Dev zfs rw,noatime,xattr,noacl 0 0 data1/r-data/Dev/CVUT /volumes/data1/r-data/Dev/CVUT zfs rw,noatime,xattr,noacl 0 0 data1/r-data/Dev/CVUT/FIT /volumes/data1/r-data/Dev/CVUT/FIT zfs rw,noatime,xattr,noacl 0 0 data1/r-data/Dev/CVUT/FIT/Pririz /volumes/data1/r-data/Dev/CVUT/FIT/Pririz zfs rw,noatime,xattr,noacl 0 0 data1/r-data/Dev/CVUT/FIT/Pririz-remote /volumes/data1/r-data/Dev/CVUT/FIT/Pririz-remote zfs rw,noatime,xattr,noacl 0 0 data1/r-data/home /volumes/data1/r-data/home zfs rw,noatime,xattr,noacl 0 0 data1/r-data/home/ola /volumes/data1/r-data/home/ola zfs rw,noatime,xattr,noacl 0 0 data1/r-data/home/saruman /volumes/data1/r-data/home/saruman zfs rw,noatime,xattr,noacl 0 0 data1/r-data/var_lib_docker /volumes/data1/r-data/var_lib_docker zfs rw,noatime,xattr,noacl 0 0 data1/zvols /volumes/data1/zvols zfs rw,noatime,xattr,noacl 0 0 data1/r-data/home /home zfs rw,noatime,xattr,noacl 0 0 data1/r-data/home/ola /home/ola zfs rw,noatime,xattr,noacl 0 0 data1/nr-data/tmp /tmp zfs rw,nosuid,nodev,noexec,xattr,noacl 0 0 data1/nr-data/usr_src /usr/src zfs rw,noatime,xattr,noacl 0 0 data1/r-data/var_lib_docker /var/lib/docker zfs rw,noatime,xattr,noacl 0 0 lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

tmpfs /run/user/1003 tmpfs rw,nosuid,nodev,relatime,size=2446936k,mode=700,uid=1003,gid=1003 0 0 data1/r-data/var_lib_docker/f55fb5b953781d7a474e92bd3bfa74be0cb6c97b1b6369bdb395951179f94c87 /var/lib/docker/zfs/graph/f55fb5b953781d7a474e92bd3bfa74be0cb6c97b1b6369bdb395951179f94c87 zfs rw,relatime,xattr,noacl 0 0 data1/r-data/var_lib_docker/f55fb5b953781d7a474e92bd3bfa74be0cb6c97b1b6369bdb395951179f94c87 /volumes/data1/r-data/var_lib_docker/zfs/graph/f55fb5b953781d7a474e92bd3bfa74be0cb6c97b1b6369bdb395951179f94c87 zfs rw,relatime,xattr,noacl 0 0 data1/r-data/var_lib_docker/833251582c5daeebd3f1ef6aea5488934881cdf279ee92f347620f96d8ae4e98 /var/lib/docker/zfs/graph/833251582c5daeebd3f1ef6aea5488934881cdf279ee92f347620f96d8ae4e98 zfs rw,relatime,xattr,noacl 0 0 data1/r-data/var_lib_docker/833251582c5daeebd3f1ef6aea5488934881cdf279ee92f347620f96d8ae4e98 /volumes/data1/r-data/var_lib_docker/zfs/graph/833251582c5daeebd3f1ef6aea5488934881cdf279ee92f347620f96d8ae4e98 zfs rw,relatime,xattr,noacl 0 0 data1/r-data/var_lib_docker/e9a68a037e59b321ae0ae95f7449d57cdaadaa759d82f8b95ef653e01c427511 /var/lib/docker/zfs/graph/e9a68a037e59b321ae0ae95f7449d57cdaadaa759d82f8b95ef653e01c427511 zfs rw,relatime,xattr,noacl 0 0 data1/r-data/var_lib_docker/e9a68a037e59b321ae0ae95f7449d57cdaadaa759d82f8b95ef653e01c427511 /volumes/data1/r-data/var_lib_docker/zfs/graph/e9a68a037e59b321ae0ae95f7449d57cdaadaa759d82f8b95ef653e01c427511 zfs rw,relatime,xattr,noacl 0 0 shm /var/lib/docker/containers/44338a047743e223ccdf4e77422938b351f771f2a20e46549b2f91ad8192485d/mounts/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 shm /volumes/data1/r-data/var_lib_docker/containers/44338a047743e223ccdf4e77422938b351f771f2a20e46549b2f91ad8192485d/mounts/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/e28783e3117c nsfs rw 0 0 shm /var/lib/docker/containers/bf07723ba97a9f92a3048432ee4a9a8436963aa3535ae728eb5c908ea97380e7/mounts/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 shm /volumes/data1/r-data/var_lib_docker/containers/bf07723ba97a9f92a3048432ee4a9a8436963aa3535ae728eb5c908ea97380e7/mounts/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 shm /var/lib/docker/containers/7e0dcb756e9b81c58ed52510aecccdc5604c544b726f5979d250a08d132d7785/mounts/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 shm /volumes/data1/r-data/var_lib_docker/containers/7e0dcb756e9b81c58ed52510aecccdc5604c544b726f5979d250a08d132d7785/mounts/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/63826128952c nsfs rw 0 0 nsfs /run/docker/netns/07370dba2d6e nsfs rw 0 0 data1/r-data/var_lib_docker/a70b4c3f94321853b4b138f1055d361eaccfa8bea284936d5aa6e49d75b9e09d /var/lib/docker/zfs/graph/a70b4c3f94321853b4b138f1055d361eaccfa8bea284936d5aa6e49d75b9e09d zfs rw,relatime,xattr,noacl 0 0 data1/r-data/var_lib_docker/a70b4c3f94321853b4b138f1055d361eaccfa8bea284936d5aa6e49d75b9e09d /volumes/data1/r-data/var_lib_docker/zfs/graph/a70b4c3f94321853b4b138f1055d361eaccfa8bea284936d5aa6e49d75b9e09d zfs rw,relatime,xattr,noacl 0 0 shm /var/lib/docker/containers/62a068a72a53ea8e56b4d7df28ca10e042aa1039cd11f44c680c05613bfa5b7b/mounts/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 shm /volumes/data1/r-data/var_lib_docker/containers/62a068a72a53ea8e56b4d7df28ca10e042aa1039cd11f44c680c05613bfa5b7b/mounts/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /run/docker/netns/38acef3a77f4 nsfs rw 0 0

Issue description

The virtual port in OpenVSwitch is not deleted after container shutdown. Subsequent container start fails, until the port is manually deleted by ovs-vsctl del-port. LXC version 3.0.2 is not affected. This issue is directly caused by commit https://github.com/lxc/lxc/commit/eed8c0ad133119972e5eda2aea132480faa7f51a

Steps to reproduce

  1. lxc-start cont-name
  2. lxc-stop cont-name
  3. lxc-start cont-name -> crash
  4. ovs-vsctl del-port cont-name
  5. lxc-start cont-name -> success

Information to attach

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist ubuntu --release bionic --arch amd64
# For additional config options, please look at lxc.container.conf(5)

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.arch = x86_64

# Container specific configuration
lxc.rootfs.path = /volumes/data1/lxc-data1/rootfs-data1/formmeup
lxc.mount.fstab = /var/lib/lxc/formmeup/fstab

lxc.uts.name = formmeup

# Network configuration
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = int-vbr0
lxc.net.0.veth.pair = formmeup
Milan-Benes commented 5 years ago

I created a kinda stupid patch to "fix" the issue:

diff -rupN lxc-3.0.3/src/lxc/nl.c lxc-3.0.3-fixed/src/lxc/nl.c
--- lxc-3.0.3/src/lxc/nl.c      2018-11-23 00:08:27.000000000 +0100
+++ lxc-3.0.3-fixed/src/lxc/nl.c        2019-02-13 11:09:32.858526997 +0100
@@ -276,7 +276,7 @@ extern int __netlink_transaction(struct
                        ret = -1;
        }

-       return ret;
+       return 0;
 }

 extern int netlink_transaction(struct nl_handler *handler,

But the question remains, why exactly is __netlink_recv ending with an error message and exiting with a non-zero exit code.

fixie-fi commented 5 years ago

I have the same problem. In Ubuntu 18.04.2 I tried LXD 3.0.3 from Ubuntu repositories and LXD 3.10 Snap package.

When I tried to include lxc.net.0.script.down in config

  raw.lxc: |-
    lxc.net.0.script.down = /usr/local/bin/ifdown-lxc

I get the following error message 

Config parsing error: Invalid config: Only interface-specific ipv4/ipv6 lxc.net. keys are allowed
stgraber commented 5 years ago

@brauner can you look into this, looks like some weird netlink behavior or something...

brauner commented 5 years ago

Can you please give me a trace log for such a failure via:

lxc-start <c> -l trace -o <c>.log
lxc-stop <c> -l trace -o <d>.log

and append both here?

nfrntrd commented 5 years ago

@brauner I'm using LXD here and didn't have the lxc-start commands loaded on my machine. I did add on my Ubuntu machine the lxc-utils, but I'm getting errors. Should there be some other command I run when I'm using the lxc command via LXD?

brauner commented 5 years ago

On Tue, Apr 02, 2019 at 09:27:14AM -0700, nfrntrd wrote:

@brauner I'm using LXD here and didn't have the lxc-start commands loaded on my machine. I did add on my Ubuntu machine the lxc-utils, but I'm getting errors. Should there be some other command I run when I'm using the lxc command via LXD?

Please start the lxd daemon with:

lxd --debug --verbose --group lxd lxc restart --force

and then do the test and the give me:

/var/log/lxd//lxc.log

brauner commented 5 years ago

@stgraber, can you give instructions how to create and retrieve verbose logs from the snap?

brauner commented 5 years ago

@nfrntrd, if you're using the snap the instructions are:

snap set lxd daemon.debug=true                                                                                                                                            systemctl reload snap.lxd.daemon

then restart the container and after you hit the bug, please do

lxc info <container-name> --show-log

and append here.

nfrntrd commented 5 years ago

@brauner Attached is my lxc.log for a container named t1 I also stopped it as well. Should be at the end of file too. Let me know what more you need and thank you! BTW - I'm using the debian packages right now (will probably switch to snap after this is issue is fixed)

Line 1194 of my log file:
lxc t1 20190402163823.748 WARN network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth0" with index 18

Is that the issue? lxc.conf.txt

lxc.log

brauner commented 5 years ago

Please also append:

lxc.conf
brauner commented 5 years ago

Are you sure this is from the correct container? This looks like it makes use of a network interface of type lxc.net.<idx>.type = phys which is a horse of a different color.

nfrntrd commented 5 years ago

Updated above with the lxc.conf.txt. Honestly, perhaps that is the issue - in LXD land all I did was use the LXD to define bridge and attach to the named bridge. If this is a configuration issue, then I apologize.

Here is how I defined it with lxc config edit t1 devices: jtest: nictype: bridged parent: jtest type: nic

brauner commented 5 years ago

Can you show the output of:

stat /sys/class/net/jtest/bridge

please.

nfrntrd commented 5 years ago

There is no file in /sys/class/net/jtest called bridge. root@saint:/sys/class/net/jtest# ls -l total 0 -r--r--r-- 1 root root 4096 Mar 20 07:23 addr_assign_type -r--r--r-- 1 root root 4096 Mar 20 07:23 address -r--r--r-- 1 root root 4096 Mar 20 07:23 addr_len -r--r--r-- 1 root root 4096 Apr 2 11:58 broadcast -rw-r--r-- 1 root root 4096 Apr 2 11:58 carrier -r--r--r-- 1 root root 4096 Apr 2 11:58 carrier_changes -r--r--r-- 1 root root 4096 Apr 2 11:58 carrier_down_count -r--r--r-- 1 root root 4096 Apr 2 11:58 carrier_up_count -r--r--r-- 1 root root 4096 Apr 2 11:58 dev_id -r--r--r-- 1 root root 4096 Apr 2 11:58 dev_port -r--r--r-- 1 root root 4096 Apr 2 11:58 dormant -r--r--r-- 1 root root 4096 Apr 2 11:58 duplex -rw-r--r-- 1 root root 4096 Apr 2 11:58 flags -rw-r--r-- 1 root root 4096 Apr 2 11:58 gro_flush_timeout -rw-r--r-- 1 root root 4096 Apr 2 11:58 ifalias -r--r--r-- 1 root root 4096 Mar 20 07:23 ifindex -r--r--r-- 1 root root 4096 Mar 20 07:23 iflink -r--r--r-- 1 root root 4096 Apr 2 11:58 link_mode -rw-r--r-- 1 root root 4096 Apr 2 11:58 mtu -r--r--r-- 1 root root 4096 Mar 20 07:23 name_assign_type -rw-r--r-- 1 root root 4096 Apr 2 11:58 netdev_group -r--r--r-- 1 root root 4096 Mar 20 07:23 operstate -r--r--r-- 1 root root 4096 Apr 2 11:58 phys_port_id -r--r--r-- 1 root root 4096 Apr 2 11:58 phys_port_name -r--r--r-- 1 root root 4096 Apr 2 11:58 phys_switch_id drwxr-xr-x 2 root root 0 Apr 2 11:58 power -rw-r--r-- 1 root root 4096 Apr 2 11:58 proto_down drwxr-xr-x 4 root root 0 Mar 20 07:23 queues -r--r--r-- 1 root root 4096 Apr 2 11:58 speed drwxr-xr-x 2 root root 0 Apr 2 11:58 statistics lrwxrwxrwx 1 root root 0 Mar 20 07:23 subsystem -> ../../../../class/net -rw-r--r-- 1 root root 4096 Apr 2 11:58 tx_queue_len -r--r--r-- 1 root root 4096 Mar 20 07:23 type -rw-r--r-- 1 root root 4096 Mar 20 07:23 uevent

brauner commented 5 years ago

Ok, that's fine. I just wanted to confirm it's an ovs bridge.