LXD 2.20 was installed over 2.0.10, no networking

[19wolf]

Required information

• Distribution: Ubuntu
• Distribution version: 16.04.2
• The output of "lxc info" or if that fails:
  • Kernel version: 4.14.0-041400-generic
  • LXD version: 2.20 -> 2.0.10
  • Storage backend in use: btrfs

Issue description

APT decided to install LXD 2.20 over LXD 2.0.10 and I didn't catch it. Containers can no longer connect to the network and I attempted to downgrade and now nothing works.

[19wolf]

Reinstalling 2.20 fixed lxd but still not networking

[19wolf]

Install created new bridge, which broke networking, attempting fix now

[19wolf]

Adjusted everything using steps from https://github.com/lxc/lxd/issues/3778 but no joy

$ lxc network list
+---------+----------+---------+-------------+---------+
|  NAME   |   TYPE   | MANAGED | DESCRIPTION | USED BY |
+---------+----------+---------+-------------+---------+
| br0     | bridge   | NO      |             | 5       |
+---------+----------+---------+-------------+---------+
| docker0 | bridge   | NO      |             | 0       |
+---------+----------+---------+-------------+---------+
| eno1    | physical | NO      |             | 0       |
+---------+----------+---------+-------------+---------+
| eno2    | physical | NO      |             | 0       |
+---------+----------+---------+-------------+---------+
$ lxc network show br0
config: {}
description: ""
name: br0
type: bridge
used_by:
- /1.0/containers/Arachne
- /1.0/containers/Clio
- /1.0/containers/Paidia
- /1.0/containers/Plutus
- /1.0/containers/Urania
managed: false
$ ifconfig br0
br0       Link encap:Ethernet  HWaddr 94:18:82:38:64:2c  
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::9618:82ff:fe38:642c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:78977 errors:0 dropped:0 overruns:0 frame:0
          TX packets:83795 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4313442 (4.3 MB)  TX bytes:232271581 (232.2 MB)

$ brctl show
bridge name bridge id       STP enabled interfaces
br0     8000.94188238642c   no      eno1
                            veth4LE0NH
                            veth6DTV46
                            veth6WL4QA
                            vethCUIQID
                            vethQ3SRP2
                            vethWM2ECV
docker0     8000.024225697b2b   no
$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet dhcp

# The primary network interface
auto br0
iface br0 inet static
    address 192.168.1.2
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 192.168.1.1

    bridge-ports eno1

[stgraber]

What does "lxc config show --expanded NAME" show for one of the containers?

[stgraber]

It sure looks like all your containers are properly bridged so it's a bit confusing that they're not getting any network configuration...

Can you also show the output of "lxc config show" (without any argument)?

[19wolf]

$ lxc config show --expanded Clio
architecture: x86_64
config:
  environment.http_proxy: ""
  image.architecture: amd64
  image.description: Debian stretch amd64 (20171003_22:42)
  image.os: Debian
  image.release: stretch
  image.serial: "20171003_22:42"
  user.network_mode: ""
  volatile.base_image: e220439d67dfa3c4176a66df9e2a234402727676d11c566add441937017adc91
  volatile.br0.hwaddr: 00:16:3e:55:60:40
  volatile.br0.name: eth1
  volatile.eth0.hwaddr: 00:16:3e:4d:7a:3a
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  br0:
    nictype: bridged
    parent: br0
    type: nic
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  media:
    path: /data/Media
    source: /data/Media
    type: disk
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
$ lxc config show
config:
  core.https_address: '[::]:8443'

[stgraber]

I think you at least want:

• lxc profile unset default environment.http_proxy
• lxc profile unset default user.network_mode
• lxc profile device remove default br0

Then post the "lxc config show --expanded NAME" output again to make sure things look good.

There's no obvious reason why your container wouldn't be getting DHCP though, unless it's freshly created and the user.network_mode key above confused the /etc/network/interfaces template, leading to broken networking in there.

[19wolf]

$ lxc profile unset default environment.http_proxy
nephele@Nephele:~$ lxc profile unset default user.network_mode
$ lxc profile device remove default br0
Device br0 removed from default
$ lxc config show --expanded Clio
architecture: x86_64
config:
  environment.http_proxy: ""
  image.architecture: amd64
  image.description: Debian stretch amd64 (20171003_22:42)
  image.os: Debian
  image.release: stretch
  image.serial: "20171003_22:42"
  user.network_mode: ""
  volatile.base_image: e220439d67dfa3c4176a66df9e2a234402727676d11c566add441937017adc91
  volatile.eth0.hwaddr: 00:16:3e:4d:7a:3a
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  media:
    path: /data/Media
    source: /data/Media
    type: disk
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

I'll try making a fresh container and will reboot

[19wolf]

My new container after a full reboot

$ lxc config show --expanded smashing-penguin
architecture: x86_64
config:
  environment.http_proxy: ""
  image.architecture: amd64
  image.description: ubuntu 16.04 LTS amd64 (release) (20171116)
  image.label: release
  image.os: ubuntu
  image.release: xenial
  image.serial: "20171116"
  image.version: "16.04"
  user.network_mode: ""
  volatile.base_image: 3927ed0cb91f768a45d45bdd261b0231918e4f672ba24d9b9f5559eba2ce4f00
  volatile.eth0.hwaddr: 00:16:3e:31:bc:93
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Still no networking

[19wolf]

If I uninstall and reinstall lxd, will I lose my containers?

[stgraber]

And there is in fact a DHCP server running somewhere on your network for that eno1 interface right?

[19wolf]

Yes, nothing has changed on my router.

[stgraber]

Oh, hold on, there is a docker bridge, that may explain some of those issues, docker has recently had the tendency to configure your kernel to block all traffic :)

Can you paste the output of "iptables -L -n -v" and "ip6tables -L -n -v"?

[19wolf]

I've had docker installed through a few reboots in the last week or two

$ sudo iptables -L -n -v
Chain INPUT (policy ACCEPT 452K packets, 160M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 1018 packets, 289K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1018  289K DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 467K packets, 1123M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1018  289K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

$ sudo ip6tables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

[19wolf]

I've removed docker (wasn't actually using it) and did iptables -F still nothing

[stgraber]

iptables -F wouldn't help as docker changed the policy :)

[stgraber]

If you look above, the FORWARD policy reads as DROP rather than the default of ACCEPT.

[stgraber]

iptables -P FORWARD ACCEPT should make things better

[19wolf]

Huzzah. Thank you!