search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.33k stars 929 forks source link

Profile removed on reboot #6139

Closed chris-sanders closed 5 years ago

chris-sanders commented 5 years ago

5318 Required information

# Issue description
I have a profile applied (lxd) to 1 container on a machine while all other containers are using a different profile (default). This machine was Juju deployed and this 1 container had it's profile changed after deployment (to the profile lxd), although I don't know why that would matter.

Most, but not all, of the time on a reboot the container has the 'lxd' profile removed and the 'default' profile set on it again. 

# Steps to reproduce

 1. Add 'lxd' profile (included below) to a container deployed by juju
 2. Remove 'default' profile (included below) from the container
 3. Reboot the host a few times and check 'lxc profile list' you'll see lxd is no longer in use and default is.

# Information to attach
 - [x] Any relevant kernel output (`dmesg`)
I'm unsure if this is relevant, but this is everything from the first to the last reference of the container in question after a reboot that triggered the profile change which I fixed by reassigning the correct profile and restarting the container.

[Sat Aug 31 21:48:10 2019] Ebtables v2.0 registered [Sat Aug 31 21:48:10 2019] audit: type=1400 audit(1567288091.674:23): apparmor="STATUS" operation="profileload" profile="unconfined" name="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>" pid=3801 comm="apparmor_parser" [Sat Aug 31 21:48:10 2019] br-bond0: port 2(veth3OSOTD) entered blocking state [Sat Aug 31 21:48:10 2019] br-bond0: port 2(veth3OSOTD) entered disabled state [Sat Aug 31 21:48:10 2019] device veth3OSOTD entered promiscuous mode [Sat Aug 31 21:48:10 2019] IPv6: ADDRCONF(NETDEV_UP): veth3OSOTD: link is not ready [Sat Aug 31 21:48:10 2019] eth0: renamed from vethIAW389 [Sat Aug 31 21:48:10 2019] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [Sat Aug 31 21:48:10 2019] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [Sat Aug 31 21:48:10 2019] IPv6: ADDRCONF(NETDEV_CHANGE): veth3OSOTD: link becomes ready [Sat Aug 31 21:48:10 2019] br-bond0: port 2(veth3OSOTD) entered blocking state [Sat Aug 31 21:48:10 2019] br-bond0: port 2(veth3OSOTD) entered forwarding state [Sat Aug 31 21:48:11 2019] audit: type=1400 audit(1567288092.398:24): apparmor="STATUS" operation="profileload" profile="unconfined" name="lxd-juju-89bcd1-4-lxd-1</var/lib/lxd>" pid=4052 comm="apparmor_parser" [Sat Aug 31 21:48:11 2019] br-bond0: port 3(vethMI29QQ) entered blocking state [Sat Aug 31 21:48:11 2019] br-bond0: port 3(vethMI29QQ) entered disabled state [Sat Aug 31 21:48:11 2019] device vethMI29QQ entered promiscuous mode [Sat Aug 31 21:48:11 2019] IPv6: ADDRCONF(NETDEV_UP): vethMI29QQ: link is not ready [Sat Aug 31 21:48:11 2019] br-bond0: port 3(vethMI29QQ) entered blocking state [Sat Aug 31 21:48:11 2019] br-bond0: port 3(vethMI29QQ) entered forwarding state [Sat Aug 31 21:48:11 2019] eth0: renamed from vethIF469I [Sat Aug 31 21:48:11 2019] IPv6: ADDRCONF(NETDEV_CHANGE): vethMI29QQ: link becomes ready [Sat Aug 31 21:48:11 2019] audit: type=1400 audit(1567288092.666:25): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="/usr/bin/lxc-start" pid=4389 comm="apparmor_parser" [Sat Aug 31 21:48:11 2019] audit: type=1400 audit(1567288092.670:26): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="/usr/bin/man" pid=4390 comm="apparmor_parser" [Sat Aug 31 21:48:11 2019] audit: type=1400 audit(1567288092.670:27): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="man_filter" pid=4390 comm="apparmor_parser" [Sat Aug 31 21:48:11 2019] audit: type=1400 audit(1567288092.670:28): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="man_groff" pid=4390 comm="apparmor_parser" [Sat Aug 31 21:48:11 2019] audit: type=1400 audit(1567288092.670:29): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="/usr/lib/snapd/snap-confine" pid=4391 comm="apparmor_parser" [Sat Aug 31 21:48:11 2019] audit: type=1400 audit(1567288092.670:30): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=4391 comm="apparmor_parser" [Sat Aug 31 21:48:11 2019] audit: type=1400 audit(1567288092.674:31): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="/usr/sbin/tcpdump" pid=4393 comm="apparmor_parser" [Sat Aug 31 21:48:11 2019] audit: type=1400 audit(1567288092.674:32): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="/sbin/dhclient" pid=4388 comm="apparmor_parser" [Sat Aug 31 21:48:11 2019] br-bond0: port 4(vethL23UIT) entered blocking state [Sat Aug 31 21:48:11 2019] br-bond0: port 4(vethL23UIT) entered disabled state [Sat Aug 31 21:48:11 2019] device vethL23UIT entered promiscuous mode [Sat Aug 31 21:48:11 2019] IPv6: ADDRCONF(NETDEV_UP): vethL23UIT: link is not ready [Sat Aug 31 21:48:12 2019] eth0: renamed from vethIO9HL4 [Sat Aug 31 21:48:12 2019] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [Sat Aug 31 21:48:12 2019] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [Sat Aug 31 21:48:12 2019] IPv6: ADDRCONF(NETDEV_CHANGE): vethL23UIT: link becomes ready [Sat Aug 31 21:48:12 2019] br-bond0: port 4(vethL23UIT) entered blocking state [Sat Aug 31 21:48:12 2019] br-bond0: port 4(vethL23UIT) entered forwarding state [Sat Aug 31 21:48:16 2019] kauditd_printk_skb: 44 callbacks suppressed [Sat Aug 31 21:48:16 2019] audit: type=1400 audit(1567288097.166:77): apparmor="STATUS" operation="profilereplace" info="same as current profile, skipping" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="/usr/bin/lxc-start" pid=6004 comm="apparmor_parser" [Sat Aug 31 21:48:16 2019] audit: type=1400 audit(1567288097.178:78): apparmor="STATUS" operation="profilereplace" info="same as current profile, skipping" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="lxc-container-default" pid=6011 comm="apparmor_parser" [Sat Aug 31 21:48:16 2019] audit: type=1400 audit(1567288097.178:79): apparmor="STATUS" operation="profilereplace" info="same as current profile, skipping" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="lxc-container-default-cgns" pid=6011 comm="apparmor_parser" [Sat Aug 31 21:48:16 2019] audit: type=1400 audit(1567288097.178:80): apparmor="STATUS" operation="profilereplace" info="same as current profile, skipping" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="lxc-container-default-with-mounting" pid=6011 comm="apparmor_parser" [Sat Aug 31 21:48:16 2019] audit: type=1400 audit(1567288097.178:81): apparmor="STATUS" operation="profilereplace" info="same as current profile, skipping" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0_:unconfined" name="lxc-container-default-with-nesting" pid=6011 comm="apparmorparser" [Sat Aug 31 21:48:16 2019] audit: type=1400 audit(1567288097.410:82): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-89bcd1-4-lxd-1</var/lib/lxd>" name="/run/systemd/unit-root/run/lock/" pid=6127 comm="(networkd)" flags="ro, nosuid, nodev, noexec, remount, bind" [Sat Aug 31 21:48:16 2019] audit: type=1400 audit(1567288097.862:83): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-89bcd1-4-lxd-2</var/lib/lxd>" name="/run/systemd/unit-root/run/lock/" pid=6584 comm="(networkd)" flags="ro, nosuid, nodev, noexec, remount, bind" [Sat Aug 31 21:48:17 2019] audit: type=1400 audit(1567288098.170:84): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-89bcd1-4-lxd-2</var/lib/lxd>" name="/home/" pid=6817 comm="(arter.sh)" flags="ro, nosuid, nodev, remount, bind" [Sat Aug 31 21:48:17 2019] audit: type=1400 audit(1567288098.234:85): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-89bcd1-4-lxd-2_</var/lib/lxd>" name="/home/" pid=6924 comm="(y-helper)" flags="ro, nosuid, nodev, remount, bind" [Sat Aug 31 21:48:18 2019] audit: type=1400 audit(1567288100.013:86): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0:unconfined" name="lxd-juju-5815ea-2</var/lib/lxd>" pid=7859 comm="apparmor_parser" [Sat Aug 31 21:48:21 2019] kauditd_printk_skb: 14 callbacks suppressed [Sat Aug 31 21:48:21 2019] audit: type=1400 audit(1567288103.111:101): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0:unconfined" name="lxd-juju-e90a83-0</var/lib/lxd>" pid=9077 comm="apparmor_parser" [Sat Aug 31 21:48:21 2019] audit: type=1400 audit(1567288103.194:102): apparmor="STATUS" operation="profileremove" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0:unconfined" name="lxd-juju-5815ea-8</var/lib/lxd>" pid=9138 comm="apparmor_parser" [Sat Aug 31 21:48:22 2019] audit: type=1400 audit(1567288104.217:103): apparmor="STATUS" operation="profileremove" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0:unconfined" name="lxd-juju-e90a83-0</var/lib/lxd>" pid=9459 comm="apparmor_parser" [Sat Aug 31 21:50:18 2019] nvme nvme0: async event result 00020101 [Sat Aug 31 21:58:46 2019] audit: type=1400 audit(1567288727.552:104): apparmor="STATUS" operation="profileload" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0:unconfined" name="lxd-juju-e90a83-0</var/lib/lxd>" pid=17712 comm="apparmor_parser" [Sat Aug 31 21:58:47 2019] audit: type=1400 audit(1567288728.392:105): apparmor="STATUS" operation="profileremove" label="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>//&:lxd-juju-89bcd1-4-lxd-0:unconfined" name="lxd-juju-e90a83-0</var/lib/lxd>" pid=17829 comm="apparmor_parser" [Sat Aug 31 21:59:26 2019] Key type ceph registered [Sat Aug 31 21:59:26 2019] libceph: loaded (mon/osd proto 15/24) [Sat Aug 31 21:59:26 2019] rbd: loaded (major 251) [Sat Aug 31 21:59:26 2019] audit: type=1400 audit(1567288768.109:106): apparmor="STATUS" operation="profilereplace" profile="unconfined" name="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>" pid=18189 comm="apparmor_parser" [Sat Aug 31 21:59:40 2019] br-bond0: port 2(veth3OSOTD) entered disabled state [Sat Aug 31 21:59:40 2019] device veth3OSOTD left promiscuous mode [Sat Aug 31 21:59:40 2019] br-bond0: port 2(veth3OSOTD) entered disabled state [Sat Aug 31 21:59:40 2019] audit: type=1400 audit(1567288781.885:107): apparmor="STATUS" operation="profileremove" profile="unconfined" name="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>" pid=18573 comm="apparmor_parser" [Sat Aug 31 21:59:43 2019] audit: type=1400 audit(1567288784.805:108): apparmor="STATUS" operation="profileload" profile="unconfined" name="lxd-juju-89bcd1-4-lxd-0</var/lib/lxd>" pid=18608 comm="apparmor_parser" [

 - [x] Container log (`lxc info NAME --show-log`)

Name: juju-89bcd1-4-lxd-0 Remote: unix:// Architecture: x86_64 Created: 2019/07/18 21:38 UTC Status: Running Type: persistent Profiles: lxd Pid: 18622 Ips: eth0: inet 192.168.0.244 vethR238KG eth0: inet6 fe80::216:3eff:fe05:2fbc vethR238KG lo: inet 127.0.0.1 lo: inet6 ::1 Resources: Processes: 1065 CPU usage: CPU usage (in seconds): 320 Memory usage: Memory (current): 3.00GB Memory (peak): 3.10GB Network usage: eth0: Bytes received: 1.28GB Bytes sent: 327.77MB Packets received: 363201 Packets sent: 369143 lo: Bytes received: 11.21kB Bytes sent: 11.21kB Packets received: 70 Packets sent: 70

Log:

lxc juju-89bcd1-4-lxd-0 20190831215945.453 WARN conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance

 - [x] Container configuration (`lxc config show NAME --expanded`)
Note the profile "lxd" is when it's fixed/working after the reboot that changes to "default" no other difference in config

architecture: x86_64 config: boot.autostart: "true" boot.autostart.delay: "60" boot.stop.priority: "10" image.architecture: amd64 image.description: ubuntu 18.04 LTS amd64 (release) (20190627.1) image.label: release image.os: ubuntu image.release: bionic image.serial: "20190627.1" image.version: "18.04" linux.kernel_modules: rbd raw.lxc: |- lxc.cgroup.devices.allow = a lxc.cap.drop= lxc.apparmor.profile = unconfined lxc.mount.auto= lxc.mount.auto=sys:rw proc:rw cgroup:rw security.nesting: "True" security.privileged: "True" user.juju-model: 6218f353-9afa-4eef-822d-549fbb89bcd1 user.network-config: | config: "disabled" user.user-data: |

cloud-config

apt_mirror: ""
bootcmd:
- install -D -m 644 /dev/null '/etc/netplan/99-juju.yaml'
- |-
  printf '%s\n' 'network:
    version: 2
    ethernets:
      eth0:
        match:
          macaddress: 00:16:3e:05:2f:bc
        addresses:
        - 192.168.0.244/23
        gateway4: 192.168.1.1
        nameservers:
          addresses: [192.168.1.1]
  ' > '/etc/netplan/99-juju.yaml'
- install -D -m 644 /dev/null '/etc/network/interfaces.templ'
- |-
  printf '%s\n' '
  auto lo {eth00_16_3e_05_2f_bc}

  iface lo inet loopback
    dns-nameservers 192.168.1.1

  iface {eth00_16_3e_05_2f_bc} inet static
    address 192.168.0.244/23
    gateway 192.168.1.1
  ' > '/etc/network/interfaces.templ'
- install -D -m 744 /dev/null '/etc/network/interfaces.py'
- |-
  printf '%s\n' 'from __future__ import print_function, unicode_literals
  import subprocess, re, argparse, os, time, shutil
  from string import Formatter

  INTERFACES_FILE="/etc/network/interfaces"
  IP_LINE = re.compile(r"^\d+: (.*?):")
  IP_HWADDR = re.compile(r".*link/ether ((\w{2}|:){11})")
  COMMAND = "ip -oneline link"
  RETRIES = 3
  WAIT = 5

  # Python3 vs Python2
  try:
      strdecode = str.decode
  except AttributeError:
      strdecode = str

  def ip_parse(ip_output):
      """parses the output of the ip command
      and returns a hwaddr->nic-name dict"""
      devices = dict()
      print("Parsing ip command output %s" % ip_output)
      for ip_line in ip_output:
          ip_line_str = strdecode(ip_line, "utf-8")
          match = IP_LINE.match(ip_line_str)
          if match is None:
              continue
          nic_name = match.group(1).split('"'"'@'"'"')[0]
          match = IP_HWADDR.match(ip_line_str)
          if match is None:
              continue
          nic_hwaddr = match.group(1)
          devices[nic_hwaddr] = nic_name
      print("Found the following devices: %s" % str(devices))
      return devices

  def replace_ethernets(interfaces_file, output_file, devices, fail_on_missing):
      """check if the contents of interfaces_file contain template
      keys corresponding to hwaddresses and replace them with
      the proper device name"""
      with open(interfaces_file + ".templ", "r") as templ_file:
          interfaces = templ_file.read()

      formatter = Formatter()
      hwaddrs = [v[1] for v in formatter.parse(interfaces) if v[1]]
      print("Found the following hwaddrs: %s" % str(hwaddrs))
      device_replacements = dict()
      for hwaddr in hwaddrs:
          hwaddr_clean = hwaddr[3:].replace("_", ":")
          if devices.get(hwaddr_clean, None):
              device_replacements[hwaddr] = devices[hwaddr_clean]
          else:
              if fail_on_missing:
                  print("Can'"'"'t find device with MAC %s, will retry" % hwaddr_clean)
                  return False
              else:
                  print("WARNING: Can'"'"'t find device with MAC %s when expected" % hwaddr_clean)
                  device_replacements[hwaddr] = hwaddr
      formatted = interfaces.format(**device_replacements)
      print("Used the values in: %s\nto fix the interfaces file:\n%s\ninto\n%s" %
             (str(device_replacements), str(interfaces), str(formatted)))

      with open(output_file, "w") as intf_out_file:
          intf_out_file.write(formatted)

      if not os.path.exists(interfaces_file + ".bak"):
          try:
              shutil.copyfile(interfaces_file, interfaces_file + ".bak")
          except OSError: #silently ignore if the file is missing
              pass
      return True

  def main():
      parser = argparse.ArgumentParser()
      parser.add_argument("--interfaces-file", dest="intf_file", default=INTERFACES_FILE)
      parser.add_argument("--output-file", dest="out_file", default=INTERFACES_FILE+".out")
      parser.add_argument("--command", default=COMMAND)
      parser.add_argument("--retries", default=RETRIES)
      parser.add_argument("--wait", default=WAIT)
      args = parser.parse_args()
      retries = int(args.retries)
      for tries in range(retries):
          ip_output = ip_parse(subprocess.check_output(args.command.split()).splitlines())
          if replace_ethernets(args.intf_file, args.out_file, ip_output, (tries != retries - 1)):
               break
          else:
               time.sleep(float(args.wait))

  if __name__ == "__main__":
      main()
  ' > '/etc/network/interfaces.py'
- |2

  if [ ! -f /sbin/ifup ]; then
      echo "No /sbin/ifup, applying netplan configuration."
      netplan generate
      netplan apply
  else
    if [ -f /usr/bin/python ]; then
        python /etc/network/interfaces.py --interfaces-file /etc/network/interfaces --output-file /etc/network/interfaces.out
    else
        python3 /etc/network/interfaces.py --interfaces-file /etc/network/interfaces --output-file /etc/network/interfaces.out
    fi
    ifdown -a
    sleep 1.5
    mv /etc/network/interfaces.out /etc/network/interfaces
    ifup -a
  fi
- |-
  printf '%s\n' 'Acquire::http::Proxy::"127.0.0.1" "DIRECT";
  Acquire::https::Proxy::"127.0.0.1" "DIRECT";
  Acquire::ftp::Proxy::"127.0.0.1" "DIRECT";
  Acquire::http::Proxy::"localhost" "DIRECT";
  Acquire::https::Proxy::"localhost" "DIRECT";
  Acquire::ftp::Proxy::"localhost" "DIRECT";
  Acquire::http::Proxy::"::1" "DIRECT";
  Acquire::https::Proxy::"::1" "DIRECT";
  Acquire::ftp::Proxy::"::1" "DIRECT";' > /etc/apt/apt.conf.d/95-juju-proxy-settings
output:
  all: '| tee -a /var/log/cloud-init-output.log'
package_update: true
package_upgrade: true
packages:
- curl
- cpu-checker
- bridge-utils
- cloud-utils
- tmux
- ubuntu-fan
runcmd:
- set -xe
- install -D -m 644 /dev/null '/etc/systemd/system/juju-clean-shutdown.service'
- |-
  printf '%s\n' '
  [Unit]
  Description=Stop all network interfaces on shutdown
  DefaultDependencies=false
  After=final.target

  [Service]
  Type=oneshot
  ExecStart=/sbin/ifdown -a -v --force
  StandardOutput=tty
  StandardError=tty

  [Install]
  WantedBy=final.target
  ' > '/etc/systemd/system/juju-clean-shutdown.service'
- /bin/systemctl enable '/etc/systemd/system/juju-clean-shutdown.service'
- install -D -m 644 /dev/null '/var/lib/juju/nonce.txt'
- printf '%s\n' 'machine-4:782e2bf5-79b8-4d01-8859-2f7a7ae6e1d6' > '/var/lib/juju/nonce.txt'
- test -n "$JUJU_PROGRESS_FD" || (exec {JUJU_PROGRESS_FD}>&2) 2>/dev/null && exec
  {JUJU_PROGRESS_FD}>&2 || JUJU_PROGRESS_FD=2
- '[ -e /etc/profile.d/juju-proxy.sh ] || printf ''\n# Added by juju\n[ -f "/etc/juju-proxy.conf"
  ] && . "/etc/juju-proxy.conf"\n'' >> /etc/profile.d/juju-proxy.sh'
- mkdir -p /var/lib/juju/locks
- (id ubuntu &> /dev/null) && chown ubuntu:ubuntu /var/lib/juju/locks
- mkdir -p /var/log/juju
- chown syslog:adm /var/log/juju
- bin='/var/lib/juju/tools/2.6.5-bionic-amd64'
- mkdir -p $bin
- echo 'Fetching Juju agent version 2.6.5 for amd64' >&$JUJU_PROGRESS_FD
- |2-

  n=1
  while true; do

      printf "Attempt $n to download agent binaries from %s...\n" 'https://192.168.0.162:17070/model/6218f353-9afa-4eef-822d-549fbb89bcd1/tools/2.6.5-bionic-amd64'
      curl -sSfw 'agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --connect-timeout 20 --noproxy "*" --insecure -o $bin/tools.tar.gz 'https://192.168.0.162:17070/model/6218f353-9afa-4eef-822d-549fbb89bcd1/tools/2.6.5-bionic-amd64' && echo "Agent binaries downloaded successfully." && break

      echo "Download failed, retrying in 15s"
      sleep 15
      n=$((n+1))
  done
- sha256sum $bin/tools.tar.gz > $bin/juju2.6.5-bionic-amd64.sha256
- grep '9cdf7800b039c57456092fb5d0a6174af61bf894f42e5f5828a1459f9d60de5f' $bin/juju2.6.5-bionic-amd64.sha256
  || (echo "Tools checksum mismatch"; exit 1)
- tar zxf $bin/tools.tar.gz -C $bin
- printf %s '{"version":"2.6.5-bionic-amd64","url":"https://192.168.0.162:17070/model/6218f353-9afa-4eef-822d-549fbb89bcd1/tools/2.6.5-bionic-amd64","sha256":"9cdf7800b039c57456092fb5d0a6174af61bf894f42e5f5828a1459f9d60de5f","size":30501516}'
  > $bin/downloaded-tools.txt
- mkdir -p '/var/lib/juju/agents/machine-4-lxd-0'
- |-
  cat > '/var/lib/juju/agents/machine-4-lxd-0/agent.conf' << 'EOF'
  # format 2.0
  tag: machine-4-lxd-0
  datadir: /var/lib/juju
  logdir: /var/log/juju
  metricsspooldir: /var/lib/juju/metricspool
  nonce: machine-4:782e2bf5-79b8-4d01-8859-2f7a7ae6e1d6
  jobs:
  - JobHostUnits
  upgradedToVersion: 2.6.5
  cacert: |
    -----BEGIN CERTIFICATE-----
    MIIDrDCCApSgAwIBAgIUBD8Z3VH3BV4GRUSKNoC6waeOHNowDQYJKoZIhvcNAQEL
    BQAwbjENMAsGA1UEChMEanVqdTEuMCwGA1UEAwwlanVqdS1nZW5lcmF0ZWQgQ0Eg
    Zm9yIG1vZGVsICJqdWp1LWNhIjEtMCsGA1UEBRMkZTM4MmIxNmQtYjA0YS00MGE5
    LTg1MDAtMDdjNjRmMjM2ZjA0MB4XDTE5MDYxMzE5NTgyM1oXDTI5MDYyMDE5NTgy
    M1owbjENMAsGA1UEChMEanVqdTEuMCwGA1UEAwwlanVqdS1nZW5lcmF0ZWQgQ0Eg
    Zm9yIG1vZGVsICJqdWp1LWNhIjEtMCsGA1UEBRMkZTM4MmIxNmQtYjA0YS00MGE5
    LTg1MDAtMDdjNjRmMjM2ZjA0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
    AQEAsZAMixw7RddHryuEy72zDrQqYQH/zCkplYZGd1vVOr+hH6UyJFdNm1TJlu/x
    c6qlpGDErVsTc4rdABHX4KsJZZvKNIg+3VDCbgaTOsbOU/AnOueDaHkjZv7uUR5e
    BYUiiSVyUUVnGXa5Q9qtQT9jXdrbg5nDFJ0hm9eMA/F8sVeWCp7nSY2hhxw0Ri9v
    kv2fgjUcjNvfskYjPGVfonfV7lPF4G5CxNWYzLOgydVAviHvHDruD1Ip+grDIyWQ
    2XG9pScuZWfZzLVaybCdqnotEsWyZWLrXQwHrxdDAgrPCy0zVdrU3fvcT4cF4jxv
    pctfzbdlCTXLBcth3PLQN8urfQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAqQwDwYD
    VR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUNrEIYYNM4bN7dsgBMqY+gOr34SQwDQYJ
    KoZIhvcNAQELBQADggEBAK6D35uJhYmlR3T/SpEsEkYO3Bxqma8Pb9aQPVczVlJu
    7vn7z0DcBanq+QbCxCAFtOXfv0FMP2TWRGrQU4JgV1IL01eW+AxZbg13Z+ebq63+
    i7IRL18jBi9rYTIlSEIz0JnZVn1pJwQ8Vsc6vWoqC3el3FnZx3inqZEck2wnAbFc
    vG16XDC2P+NR0ADUWgbND7AkZlUH1qoY+Hy8KKuxm8rv6IUgQO0+jpqwtC30G3Tm
    XmjtnxD16V1wcctHfuMZByqzccU31/PWFZroEeKUjXFJh0zxDEBKwZ1+Pk/kYCnX
    RGNI59idxSqbI04nPWbVchz+qll6Ah/LsxdJbb9kNhk=
    -----END CERTIFICATE-----
  controller: controller-39e73c1d-47a2-4d1e-8d39-9d72d67f9fd7
  model: model-6218f353-9afa-4eef-822d-549fbb89bcd1
  apiaddresses:
  - 192.168.0.162:17070
  oldpassword: REDACTED
  values:
    AGENT_SERVICE_NAME: jujud-machine-4-lxd-0
    CONTAINER_TYPE: lxd
    PROVIDER_TYPE: maas
  mongoversion: "0.0"

  EOF
- chmod 0600 '/var/lib/juju/agents/machine-4-lxd-0/agent.conf'
- ln -s 2.6.5-bionic-amd64 '/var/lib/juju/tools/machine-4-lxd-0'
- echo 'Starting Juju machine agent (service jujud-machine-4-lxd-0)' >&$JUJU_PROGRESS_FD
- mkdir -p '/lib/systemd/system/jujud-machine-4-lxd-0'
- |-
  cat > '/lib/systemd/system/jujud-machine-4-lxd-0/exec-start.sh' << 'EOF'
  #!/usr/bin/env bash

  # Set up logging.
  touch '/var/log/juju/machine-4-lxd-0.log'
  chown syslog:syslog '/var/log/juju/machine-4-lxd-0.log'
  chmod 0600 '/var/log/juju/machine-4-lxd-0.log'
  exec >> '/var/log/juju/machine-4-lxd-0.log'
  exec 2>&1

  # Run the script.
  '/var/lib/juju/tools/machine-4-lxd-0/jujud' machine --data-dir '/var/lib/juju' --machine-id 4/lxd/0 --debug
  EOF
- chmod 0755 '/lib/systemd/system/jujud-machine-4-lxd-0/exec-start.sh'
- |-
  cat > '/lib/systemd/system/jujud-machine-4-lxd-0/jujud-machine-4-lxd-0.service' << 'EOF'
  [Unit]
  Description=juju agent for machine-4-lxd-0
  After=syslog.target
  After=network.target
  After=systemd-user-sessions.service

  [Service]
  LimitNOFILE=64000
  ExecStart=/lib/systemd/system/jujud-machine-4-lxd-0/exec-start.sh
  Restart=on-failure
  TimeoutSec=300

  [Install]
  WantedBy=multi-user.target

  EOF
- /bin/systemctl link '/lib/systemd/system/jujud-machine-4-lxd-0/jujud-machine-4-lxd-0.service'
- /bin/systemctl daemon-reload
- /bin/systemctl enable '/lib/systemd/system/jujud-machine-4-lxd-0/jujud-machine-4-lxd-0.service'
- /bin/systemctl start jujud-machine-4-lxd-0.service
- rm $bin/tools.tar.gz && rm $bin/juju2.6.5-bionic-amd64.sha256
- ifconfig
users:
- groups:
  - adm
  - audio
  - cdrom
  - dialout
  - dip
  - floppy
  - netdev
  - plugdev
  - sudo
  - video
  lock_passwd: true
  name: ubuntu
  shell: /bin/bash
  ssh-authorized-keys:
  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCaLXFR1bZIufdEMhJ/dSgX6VWo4OHI2nb2MZu61hyxnC6ynaITWCxGwBFFBduilIixBZ27TQZTIw5rCeh1jMXlAshTGT/W0hTIs85NO3/bZ6xtj8FXFTcLLaZD4zm8ZiSULAggTQYP5UrjkwoDuIWYwR0ZPm2PwNAu/0cUydkr8eeUwJiWarTKjugaVufT85vC+H9AzypviSBQaIeU1kXA+ny1FvnHm7+mFWCDG1Iyky4uP9A4vumt93EyS7WVmirJ6IranvLDmwwQKrEjcyGUxbn+hZ1lf0HKzKzJmuBWiU1+4DNXey9SLeru2AMETHahGSbgqT8P6ELMcKhjMi1/
    Juju:juju-client-key
  - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDEjX1LfTNmy2kd4jhSWliNGkkDzBebyrwDnHq1DMG8UebN1aVms7RFJFZQLmFo5L4fxfVgu9YAQxKa7bPbHH8+TWTPuqN1mC9d2XCy9cSGa/28/rslxh+ex+Kn4DejriGiEgJjrtTjDfOB+FxBb64oaudA/yPksENcw51ycB74fshPhP+XnQadqt8WD7HmMUk8YfH5QzDfKgPpCcZV5eHjgVfL9ritUANK6swCycHhZoo+wFG3pDXW1eQorKB+oT5FK6Rwk9QE0kPW52CpyYSstj5DBJK6ZGBM+I8m8hxR5km6G1h/Yh3/d2D3tubq0qAxLYzAO7Gkq+PbJR0ij1H2aooKFbNb3y5fMSvRI6wcty2ZkubtBKUUz93CUlAR3ZOLcEL6Mspa8ICak7ay0P4BtQIoysGSvEBBX158e4wE0r0dJRBA3C3mnGDTsqTCFd/CZOWH1PaMUjEaCPyCSwIp6fB8VtfeRM/lgwNsJoIljgzqgdfhgyKiiHUQJM4Vsoz65Ud7vvFv7uoqL7gYHgunoz5FjDluBoOF3Y2egcTkXnPvON2VJyPuutNVnWGAfWrnAQOORItmk1G7GIqN9xKnA8fXfWMfCPbkE+YLJRJkttrhoOppwdjROPpzS04i8jYHpQ7rFsvtgKpfo+QExzyp7AmIkme5+ba1LHNBbBjuGQ==
    Juju:chris@chris-XPS-15-9550
  sudo:
  - ALL=(ALL) NOPASSWD:ALL

volatile.base_image: 6ae1c6e92017402f1aee655fa8d785ee9d2337a3369d76115cecad5e7a303e07 volatile.idmap.base: "0" volatile.idmap.next: '[]' volatile.last_state.idmap: '[]' volatile.last_state.power: RUNNING devices: eth0: hwaddr: 00:16:3e:05:2f:bc mtu: "9000" name: eth0 nictype: bridged parent: br-bond0 type: nic root: path: / pool: default type: disk ephemeral: false profiles:

t=2019-08-31T10:21:26+0000 lvl=info msg="Updating images" 
t=2019-08-31T10:21:26+0000 lvl=info msg="Done updating images" 
t=2019-08-31T16:21:26+0000 lvl=info msg="Updating images" 
t=2019-08-31T16:21:26+0000 lvl=info msg="Done updating images" 
t=2019-08-31T21:43:24+0000 lvl=info msg="Asked to shutdown by API, shutting down containers" 
t=2019-08-31T21:43:24+0000 lvl=info msg="Shutting down container" action=shutdown created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-0 timeout=30s used=2019-08-24T22:59:37+0000
t=2019-08-31T21:43:24+0000 lvl=info msg="Shutting down container" action=shutdown created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-1 timeout=30s used=2019-08-24T22:21:26+0000
t=2019-08-31T21:43:24+0000 lvl=info msg="Shutting down container" action=shutdown created=2019-07-21T19:18:11+0000 ephemeral=false name=juju-89bcd1-4-lxd-2 timeout=30s used=2019-08-24T22:21:26+0000
t=2019-08-31T21:43:25+0000 lvl=info msg="Shut down container" action=shutdown created=2019-07-21T19:18:11+0000 ephemeral=false name=juju-89bcd1-4-lxd-2 timeout=30s used=2019-08-24T22:21:26+0000
t=2019-08-31T21:43:28+0000 lvl=info msg="Shut down container" action=shutdown created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-1 timeout=30s used=2019-08-24T22:21:26+0000
t=2019-08-31T21:43:41+0000 lvl=info msg="Shut down container" action=shutdown created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-0 timeout=30s used=2019-08-24T22:59:37+0000
t=2019-08-31T21:43:41+0000 lvl=info msg="Starting shutdown sequence" 
t=2019-08-31T21:43:41+0000 lvl=info msg="Stopping REST API handler:" 
t=2019-08-31T21:43:41+0000 lvl=info msg=" - closing socket" socket=/var/lib/lxd/unix.socket
t=2019-08-31T21:43:41+0000 lvl=info msg="Stopping /dev/lxd handler" 
t=2019-08-31T21:43:41+0000 lvl=info msg=" - closing socket" socket=/var/lib/lxd/devlxd/sock
t=2019-08-31T21:43:41+0000 lvl=info msg="Closing the database" 
t=2019-08-31T21:43:41+0000 lvl=info msg="Stopping REST API handler:" 
t=2019-08-31T21:43:41+0000 lvl=info msg="Stopping /dev/lxd handler" 
t=2019-08-31T21:43:41+0000 lvl=info msg="Stopping REST API handler:" 
t=2019-08-31T21:43:41+0000 lvl=info msg="Stopping /dev/lxd handler" 
t=2019-08-31T21:43:41+0000 lvl=info msg="Unmounting temporary filesystems" 
t=2019-08-31T21:43:41+0000 lvl=info msg="Done unmounting temporary filesystems" 
t=2019-08-31T21:43:41+0000 lvl=info msg="Saving simplestreams cache" 
t=2019-08-31T21:43:41+0000 lvl=info msg="Saved simplestreams cache" 
t=2019-08-31T21:48:05+0000 lvl=info msg="LXD 3.0.3 is starting in normal mode" path=/var/lib/lxd
t=2019-08-31T21:48:05+0000 lvl=info msg="Kernel uid/gid map:" 
t=2019-08-31T21:48:05+0000 lvl=info msg=" - u 0 0 4294967295" 
t=2019-08-31T21:48:05+0000 lvl=info msg=" - g 0 0 4294967295" 
t=2019-08-31T21:48:05+0000 lvl=info msg="Configured LXD uid/gid map:" 
t=2019-08-31T21:48:05+0000 lvl=info msg=" - u 0 100000 65536" 
t=2019-08-31T21:48:05+0000 lvl=info msg=" - g 0 100000 65536" 
t=2019-08-31T21:48:05+0000 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." 
t=2019-08-31T21:48:05+0000 lvl=info msg="Kernel features:" 
t=2019-08-31T21:48:05+0000 lvl=info msg=" - netnsid-based network retrieval: no" 
t=2019-08-31T21:48:05+0000 lvl=info msg=" - unprivileged file capabilities: yes" 
t=2019-08-31T21:48:05+0000 lvl=info msg="Initializing local database" 
t=2019-08-31T21:48:06+0000 lvl=info msg="Starting /dev/lxd handler:" 
t=2019-08-31T21:48:06+0000 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2019-08-31T21:48:06+0000 lvl=info msg="REST API daemon:" 
t=2019-08-31T21:48:06+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/lib/lxd/unix.socket
t=2019-08-31T21:48:06+0000 lvl=info msg="Initializing global database" 
t=2019-08-31T21:48:06+0000 lvl=info msg="Initializing storage pools" 
t=2019-08-31T21:48:06+0000 lvl=info msg="Initializing networks" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Pruning leftover image files" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Done pruning leftover image files" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Loading daemon configuration" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Pruning expired images" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Done pruning expired images" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Expiring log files" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Done expiring log files" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Updating instance types" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Done updating instance types" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Updating images" 
t=2019-08-31T21:48:07+0000 lvl=info msg="Done updating images" 
t=2019-08-31T21:48:11+0000 lvl=info msg="Starting container" action=start created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-0 stateful=false used=2019-08-24T22:59:37+0000
t=2019-08-31T21:48:12+0000 lvl=info msg="Started container" action=start created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-0 stateful=false used=2019-08-24T22:59:37+0000
t=2019-08-31T21:48:12+0000 lvl=info msg="Starting container" action=start created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-1 stateful=false used=2019-08-24T22:21:26+0000
t=2019-08-31T21:48:12+0000 lvl=info msg="Started container" action=start created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-1 stateful=false used=2019-08-24T22:21:26+0000
t=2019-08-31T21:48:12+0000 lvl=info msg="Starting container" action=start created=2019-07-21T19:18:11+0000 ephemeral=false name=juju-89bcd1-4-lxd-2 stateful=false used=2019-08-24T22:21:26+0000
t=2019-08-31T21:48:13+0000 lvl=info msg="Started container" action=start created=2019-07-21T19:18:11+0000 ephemeral=false name=juju-89bcd1-4-lxd-2 stateful=false used=2019-08-24T22:21:26+0000
t=2019-08-31T21:59:39+0000 lvl=info msg="Shutting down container" action=shutdown created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-0 timeout=-1s used=2019-08-31T21:48:11+0000
t=2019-08-31T21:59:41+0000 lvl=info msg="Shut down container" action=shutdown created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-0 timeout=-1s used=2019-08-31T21:48:11+0000
t=2019-08-31T21:59:44+0000 lvl=info msg="Starting container" action=start created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-0 stateful=false used=2019-08-31T21:48:11+0000
t=2019-08-31T21:59:45+0000 lvl=info msg="Started container" action=start created=2019-07-18T21:38:57+0000 ephemeral=false name=juju-89bcd1-4-lxd-0 stateful=false used=2019-08-31T21:48:11+0000

Not sure how I could include either of the above since the reproduction happens during boot.

The Profiles in question, when 4-lxd-0 is on the correct profile.

config:
  boot.autostart.priority: "10"
description: Default LXD profile
devices:
  eth0:
    mtu: "9000"
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/containers/juju-89bcd1-4-lxd-1
- /1.0/containers/juju-89bcd1-4-lxd-2
config:
  boot.autostart.delay: "60"
  boot.stop.priority: "10"
  linux.kernel_modules: rbd
  raw.lxc: |-
    lxc.cgroup.devices.allow = a
    lxc.cap.drop=
    lxc.apparmor.profile = unconfined
    lxc.mount.auto=
    lxc.mount.auto=sys:rw proc:rw cgroup:rw
  security.nesting: "True"
  security.privileged: "True"
  user.network-config: |
    version: 1
    config:
      - type: physical
        name: eth0
      - type: bridge
        name: br0
        bridge_interfaces:
          - eth0
        subnets:
          - type: dhcp
description: LXD Cluster profile
devices:
  eth0:
    mtu: "9000"
    nictype: bridged
    parent: br-bond0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: lxd
used_by:
- /1.0/containers/juju-89bcd1-4-lxd-0
stgraber commented 5 years ago

LXD doesn't have any code that can alter containers in such a way.

So it's either filesystem corruption causing your LXD database to not be written to disk properly and so the change never persisting or it's Juju resetting the container config on startup. The latter seems more likely to me

chris-sanders commented 5 years ago

Ok thanks, I'll check in with the juju team to see what could be causing this.

stgraber commented 5 years ago

Thanks, let us know if we can help with anything. If you need proof of what's going on, starting LXD with --debug would show all API queries, that should include PUT calls against /1.0/containers/NAME causing those configuration changes.