search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 929 forks source link

LXD-3.4 throwing 500 Internal Server Error #4944

Closed davidfavor closed 6 years ago

davidfavor commented 6 years ago

Having this problem on one machine of many...

Any suggestions on how to recover, without a reboot, will be great.

If any other data required, just let me know.

Thanks.

net11 # lxc config unset core.https_address
Error: Failed to fetch http://unix.socket/1.0: 500 Internal Server Error

net11 # snap refresh
All snaps up to date.

net11 # snap restart lxd
Restarted.

net11 # lxc config unset core.https_address
Error: Failed to fetch http://unix.socket/1.0: 500 Internal Server Error

net11 # which lxc
/snap/bin/lxc

net11 # lxc --version
3.4

net11 # dpkg -l | egrep -e lxc -e lxd
net11 #
stgraber commented 6 years ago

@davidfavor can you check that it's still an issue and immediately show snap list so we can be sure what revision you're on?

davidfavor commented 6 years ago 
net11 # snap list
Name  Version    Rev   Tracking  Publisher  Notes
core  16-2.34.3  5145  stable    canonical  core
lxd   3.4        8349  stable    canonical  -
stgraber commented 6 years ago

Ok, so that's indeed up to date.

Can you show ps fauxww?

davidfavor commented 6 years ago 
net11 # ps fauxww
USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
... ... ...
root     412060  0.7  0.0 3620084 40016 ?       Sl   03:07   5:07 lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd
lxd      412294  0.0  0.0  49984  2652 ?        S    03:07   0:04 dnsmasq --strict-order --bind-interfaces --pid-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.pid --except-interface=lo --interface=lxdbr0 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.72.34.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.72.34.2,10.72.34.254,1h --listen-address=fd42:1aa4:cd58:d70d::1 --enable-ra --dhcp-range ::,constructor:lxdbr0,ra-stateless,ra-names -s lxd -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd
root     691560  0.0  0.0   4504  1024 ?        S    14:37   0:00 /bin/sh /snap/lxd/8349/commands/daemon.start
root     691609  0.0  0.0   4376   704 ?        S    14:37   0:00  \_ sleep 1
root     691610  0.0  0.0   4504  1660 ?        Ss   14:37   0:00 /bin/sh /snap/lxd/8349/commands/daemon.start
davidfavor commented 6 years ago

Maybe this helps... which just started... sometime in last few hours...

net11 # lxd exec net11-dns bash
WARN[08-17|14:38:34] CGroup memory swap accounting is disabled, swap limits will be ignored. 

EROR[08-17|14:38:39] Failed to start the daemon: Failed to create raft factory: failed to create bolt store for raft logs: timeout 
Error: Failed to create raft factory: failed to create bolt store for raft logs: timeout
stgraber commented 6 years ago

Run:

That should take care of it.

davidfavor commented 6 years ago

Yes. That did it... so...

Kill the master (whatever correct term might be) LXD process + restart it, if I understand the ps output.

Let me know if this is correct.

davidfavor commented 6 years ago

Also, if you've seen this before... it this situation can be avoided, let me know how.

Thanks.

stgraber commented 6 years ago

Yeah, that process didn't get properly cleaned up during the upgrade, I'm looking at changing our logic in the startup script to catch that.

stgraber commented 6 years ago

https://discuss.linuxcontainers.org/t/status-on-issues-with-lxd-3-4-snap/2479/2

davidfavor commented 6 years ago

I see, so sometimes... the snap upgrade for 3.4 leaves 3.4 out to lunch somewhere.

Thanks.

davidfavor commented 6 years ago

I'm just glad there's a simple way to fix this without a reboot.

Thanks for your super fast assistance!

davidfavor commented 6 years ago

The update last night from LXD-3.4.8349 to LXD-3.4.8393 caused same problem on same machine + same problem on other machines.

Unsure if this problem still remains or if it should have been fixed.

Just reporting the news.

The above fix clears the error on all machines I've tested so far.

stgraber commented 6 years ago

That's odd, 8393 has the fix which should locate and kill any leftover LXD processes...

Can you show journalctl -u snap.lxd.daemon -n500 on a system where the 8393 upgrade still required manual intervention?

davidfavor commented 6 years ago

Yep. Getting very bad now. Problem arises repeatedly over time. Every few hours, I have to do the fix above.

davidfavor commented 6 years ago 
net11 # journalctl --no-pager -u snap.lxd.daemon -n500
-- Logs begin at Sun 2018-07-08 09:07:15 CDT, end at Sat 2018-08-18 11:13:00 CDT. --
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: => Preparing the system
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: ==> Loading snap configuration
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: ==> Setting up kmod wrapper
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: ==> Preparing /boot
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: ==> Preparing a clean copy of /run
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: ==> Preparing a clean copy of /etc
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: ==> Setting up ceph configuration
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: ==> Setting up LVM configuration
Aug 17 14:41:10 net11.bizcooker.com lxd.daemon[697321]: ==> Rotating logs
Aug 17 14:41:12 net11.bizcooker.com lxd.daemon[697321]: ==> Setting up ZFS (0.7)
Aug 17 14:41:12 net11.bizcooker.com lxd.daemon[697321]: ==> Escaping the systemd cgroups
Aug 17 14:41:12 net11.bizcooker.com lxd.daemon[697321]: ==> Escaping the systemd process resource limits
Aug 17 14:41:12 net11.bizcooker.com lxd.daemon[697321]: => Re-using existing LXCFS
Aug 17 14:41:12 net11.bizcooker.com lxd.daemon[697321]: => Starting LXD
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697321]: => LXD is ready
Aug 17 14:41:13 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:13 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:13 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:13 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 121.
Aug 17 14:41:13 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:13 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: => Preparing the system
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Loading snap configuration
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Setting up kmod wrapper
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Preparing /boot
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Preparing a clean copy of /run
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Preparing a clean copy of /etc
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Setting up ceph configuration
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Setting up LVM configuration
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Rotating logs
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Setting up ZFS (0.7)
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Escaping the systemd cgroups
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: ==> Escaping the systemd process resource limits
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: => Re-using existing LXCFS
Aug 17 14:41:13 net11.bizcooker.com lxd.daemon[697426]: => Starting LXD
Aug 17 14:41:14 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:14 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:15 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:15 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 122.
Aug 17 14:41:15 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:15 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: => Preparing the system
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Loading snap configuration
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Setting up kmod wrapper
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Preparing /boot
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Preparing a clean copy of /run
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Preparing a clean copy of /etc
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Setting up ceph configuration
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Setting up LVM configuration
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Rotating logs
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Setting up ZFS (0.7)
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Escaping the systemd cgroups
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: ==> Escaping the systemd process resource limits
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: => Re-using existing LXCFS
Aug 17 14:41:15 net11.bizcooker.com lxd.daemon[697531]: => Starting LXD
Aug 17 14:41:17 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:17 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:17 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:17 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 123.
Aug 17 14:41:17 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:17 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: => Preparing the system
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Loading snap configuration
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Setting up kmod wrapper
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Preparing /boot
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Preparing a clean copy of /run
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Preparing a clean copy of /etc
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Setting up ceph configuration
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Setting up LVM configuration
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Rotating logs
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Setting up ZFS (0.7)
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Escaping the systemd cgroups
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: ==> Escaping the systemd process resource limits
Aug 17 14:41:17 net11.bizcooker.com lxd.daemon[697636]: => Re-using existing LXCFS
Aug 17 14:41:18 net11.bizcooker.com lxd.daemon[697636]: => Starting LXD
Aug 17 14:41:19 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:19 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:19 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:19 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 124.
Aug 17 14:41:19 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:19 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: => Preparing the system
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: ==> Loading snap configuration
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: ==> Setting up kmod wrapper
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: ==> Preparing /boot
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: ==> Preparing a clean copy of /run
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: ==> Preparing a clean copy of /etc
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: ==> Setting up ceph configuration
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: ==> Setting up LVM configuration
Aug 17 14:41:19 net11.bizcooker.com lxd.daemon[697740]: ==> Rotating logs
Aug 17 14:41:20 net11.bizcooker.com lxd.daemon[697740]: ==> Setting up ZFS (0.7)
Aug 17 14:41:20 net11.bizcooker.com lxd.daemon[697740]: ==> Escaping the systemd cgroups
Aug 17 14:41:20 net11.bizcooker.com lxd.daemon[697740]: ==> Escaping the systemd process resource limits
Aug 17 14:41:20 net11.bizcooker.com lxd.daemon[697740]: => Re-using existing LXCFS
Aug 17 14:41:20 net11.bizcooker.com lxd.daemon[697740]: => Starting LXD
Aug 17 14:41:22 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:22 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:22 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:22 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 125.
Aug 17 14:41:22 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:22 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: => Preparing the system
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Loading snap configuration
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Setting up kmod wrapper
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Preparing /boot
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Preparing a clean copy of /run
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Preparing a clean copy of /etc
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Setting up ceph configuration
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Setting up LVM configuration
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Rotating logs
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Setting up ZFS (0.7)
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Escaping the systemd cgroups
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: ==> Escaping the systemd process resource limits
Aug 17 14:41:22 net11.bizcooker.com lxd.daemon[697845]: => Re-using existing LXCFS
Aug 17 14:41:23 net11.bizcooker.com lxd.daemon[697845]: => Starting LXD
Aug 17 14:41:24 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:24 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:24 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:24 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 126.
Aug 17 14:41:24 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:24 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:24 net11.bizcooker.com lxd.daemon[697952]: => Preparing the system
Aug 17 14:41:24 net11.bizcooker.com lxd.daemon[697952]: ==> Loading snap configuration
Aug 17 14:41:24 net11.bizcooker.com lxd.daemon[697952]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:24 net11.bizcooker.com lxd.daemon[697952]: ==> Setting up kmod wrapper
Aug 17 14:41:24 net11.bizcooker.com lxd.daemon[697952]: ==> Preparing /boot
Aug 17 14:41:24 net11.bizcooker.com lxd.daemon[697952]: ==> Preparing a clean copy of /run
Aug 17 14:41:25 net11.bizcooker.com lxd.daemon[697952]: ==> Preparing a clean copy of /etc
Aug 17 14:41:25 net11.bizcooker.com lxd.daemon[697952]: ==> Setting up ceph configuration
Aug 17 14:41:25 net11.bizcooker.com lxd.daemon[697952]: ==> Setting up LVM configuration
Aug 17 14:41:25 net11.bizcooker.com lxd.daemon[697952]: ==> Rotating logs
Aug 17 14:41:25 net11.bizcooker.com lxd.daemon[697952]: ==> Setting up ZFS (0.7)
Aug 17 14:41:25 net11.bizcooker.com lxd.daemon[697952]: ==> Escaping the systemd cgroups
Aug 17 14:41:25 net11.bizcooker.com lxd.daemon[697952]: ==> Escaping the systemd process resource limits
Aug 17 14:41:25 net11.bizcooker.com lxd.daemon[697952]: => Re-using existing LXCFS
Aug 17 14:41:25 net11.bizcooker.com lxd.daemon[697952]: => Starting LXD
Aug 17 14:41:26 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:26 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:26 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:26 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 127.
Aug 17 14:41:26 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:26 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: => Preparing the system
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: ==> Loading snap configuration
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: ==> Setting up kmod wrapper
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: ==> Preparing /boot
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: ==> Preparing a clean copy of /run
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: ==> Preparing a clean copy of /etc
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: ==> Setting up ceph configuration
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: ==> Setting up LVM configuration
Aug 17 14:41:26 net11.bizcooker.com lxd.daemon[698055]: ==> Rotating logs
Aug 17 14:41:27 net11.bizcooker.com lxd.daemon[698055]: ==> Setting up ZFS (0.7)
Aug 17 14:41:27 net11.bizcooker.com lxd.daemon[698055]: ==> Escaping the systemd cgroups
Aug 17 14:41:27 net11.bizcooker.com lxd.daemon[698055]: ==> Escaping the systemd process resource limits
Aug 17 14:41:28 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=137/n/a
Aug 17 14:41:28 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:29 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:29 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 128.
Aug 17 14:41:29 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:29 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: => Preparing the system
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Loading snap configuration
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Setting up kmod wrapper
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Preparing /boot
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Preparing a clean copy of /run
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Preparing a clean copy of /etc
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Setting up ceph configuration
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Setting up LVM configuration
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Rotating logs
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Setting up ZFS (0.7)
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Escaping the systemd cgroups
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: ==> Escaping the systemd process resource limits
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: => Re-using existing LXCFS
Aug 17 14:41:29 net11.bizcooker.com lxd.daemon[698162]: => Starting LXD
Aug 17 14:41:31 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:31 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:31 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:31 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 129.
Aug 17 14:41:31 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:31 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: => Preparing the system
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: ==> Loading snap configuration
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: ==> Setting up kmod wrapper
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: ==> Preparing /boot
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: ==> Preparing a clean copy of /run
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: ==> Preparing a clean copy of /etc
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: ==> Setting up ceph configuration
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: ==> Setting up LVM configuration
Aug 17 14:41:31 net11.bizcooker.com lxd.daemon[698270]: ==> Rotating logs
Aug 17 14:41:32 net11.bizcooker.com lxd.daemon[698270]: ==> Setting up ZFS (0.7)
Aug 17 14:41:32 net11.bizcooker.com lxd.daemon[698270]: ==> Escaping the systemd cgroups
Aug 17 14:41:32 net11.bizcooker.com lxd.daemon[698270]: ==> Escaping the systemd process resource limits
Aug 17 14:41:32 net11.bizcooker.com lxd.daemon[698270]: => Re-using existing LXCFS
Aug 17 14:41:32 net11.bizcooker.com lxd.daemon[698270]: => Starting LXD
Aug 17 14:41:33 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:33 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:33 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:33 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 130.
Aug 17 14:41:33 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:33 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: => Preparing the system
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: ==> Loading snap configuration
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: ==> Setting up kmod wrapper
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: ==> Preparing /boot
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: ==> Preparing a clean copy of /run
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: ==> Preparing a clean copy of /etc
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: ==> Setting up ceph configuration
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: ==> Setting up LVM configuration
Aug 17 14:41:33 net11.bizcooker.com lxd.daemon[698373]: ==> Rotating logs
Aug 17 14:41:34 net11.bizcooker.com lxd.daemon[698373]: ==> Setting up ZFS (0.7)
Aug 17 14:41:34 net11.bizcooker.com lxd.daemon[698373]: ==> Escaping the systemd cgroups
Aug 17 14:41:34 net11.bizcooker.com lxd.daemon[698373]: ==> Escaping the systemd process resource limits
Aug 17 14:41:34 net11.bizcooker.com lxd.daemon[698373]: => Re-using existing LXCFS
Aug 17 14:41:34 net11.bizcooker.com lxd.daemon[698373]: => Starting LXD
Aug 17 14:41:35 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:35 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:35 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:35 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 131.
Aug 17 14:41:35 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:35 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:35 net11.bizcooker.com lxd.daemon[698486]: => Preparing the system
Aug 17 14:41:35 net11.bizcooker.com lxd.daemon[698486]: ==> Loading snap configuration
Aug 17 14:41:35 net11.bizcooker.com lxd.daemon[698486]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:35 net11.bizcooker.com lxd.daemon[698486]: ==> Setting up kmod wrapper
Aug 17 14:41:35 net11.bizcooker.com lxd.daemon[698486]: ==> Preparing /boot
Aug 17 14:41:35 net11.bizcooker.com lxd.daemon[698486]: ==> Preparing a clean copy of /run
Aug 17 14:41:35 net11.bizcooker.com lxd.daemon[698486]: ==> Preparing a clean copy of /etc
Aug 17 14:41:36 net11.bizcooker.com lxd.daemon[698486]: ==> Setting up ceph configuration
Aug 17 14:41:36 net11.bizcooker.com lxd.daemon[698486]: ==> Setting up LVM configuration
Aug 17 14:41:36 net11.bizcooker.com lxd.daemon[698486]: ==> Rotating logs
Aug 17 14:41:36 net11.bizcooker.com lxd.daemon[698486]: ==> Setting up ZFS (0.7)
Aug 17 14:41:36 net11.bizcooker.com lxd.daemon[698486]: ==> Escaping the systemd cgroups
Aug 17 14:41:36 net11.bizcooker.com lxd.daemon[698486]: ==> Escaping the systemd process resource limits
Aug 17 14:41:36 net11.bizcooker.com lxd.daemon[698486]: => Re-using existing LXCFS
Aug 17 14:41:36 net11.bizcooker.com lxd.daemon[698486]: => Starting LXD
Aug 17 14:41:38 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:38 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:38 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:38 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 132.
Aug 17 14:41:38 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:38 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:38 net11.bizcooker.com lxd.daemon[698596]: => Preparing the system
Aug 17 14:41:38 net11.bizcooker.com lxd.daemon[698596]: ==> Loading snap configuration
Aug 17 14:41:38 net11.bizcooker.com lxd.daemon[698596]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:38 net11.bizcooker.com lxd.daemon[698596]: ==> Setting up kmod wrapper
Aug 17 14:41:38 net11.bizcooker.com lxd.daemon[698596]: ==> Preparing /boot
Aug 17 14:41:38 net11.bizcooker.com lxd.daemon[698596]: ==> Preparing a clean copy of /run
Aug 17 14:41:39 net11.bizcooker.com lxd.daemon[698596]: ==> Preparing a clean copy of /etc
Aug 17 14:41:39 net11.bizcooker.com lxd.daemon[698596]: ==> Setting up ceph configuration
Aug 17 14:41:39 net11.bizcooker.com lxd.daemon[698596]: ==> Setting up LVM configuration
Aug 17 14:41:39 net11.bizcooker.com lxd.daemon[698596]: ==> Rotating logs
Aug 17 14:41:39 net11.bizcooker.com lxd.daemon[698596]: ==> Setting up ZFS (0.7)
Aug 17 14:41:39 net11.bizcooker.com lxd.daemon[698596]: ==> Escaping the systemd cgroups
Aug 17 14:41:39 net11.bizcooker.com lxd.daemon[698596]: ==> Escaping the systemd process resource limits
Aug 17 14:41:39 net11.bizcooker.com lxd.daemon[698596]: => Re-using existing LXCFS
Aug 17 14:41:39 net11.bizcooker.com lxd.daemon[698596]: => Starting LXD
Aug 17 14:41:41 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:41 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:41 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:41 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 133.
Aug 17 14:41:41 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:41 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: => Preparing the system
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: ==> Loading snap configuration
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: ==> Setting up kmod wrapper
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: ==> Preparing /boot
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: ==> Preparing a clean copy of /run
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: ==> Preparing a clean copy of /etc
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: ==> Setting up ceph configuration
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: ==> Setting up LVM configuration
Aug 17 14:41:41 net11.bizcooker.com lxd.daemon[698701]: ==> Rotating logs
Aug 17 14:41:42 net11.bizcooker.com lxd.daemon[698701]: ==> Setting up ZFS (0.7)
Aug 17 14:41:42 net11.bizcooker.com lxd.daemon[698701]: ==> Escaping the systemd cgroups
Aug 17 14:41:42 net11.bizcooker.com lxd.daemon[698701]: ==> Escaping the systemd process resource limits
Aug 17 14:41:42 net11.bizcooker.com lxd.daemon[698701]: => Re-using existing LXCFS
Aug 17 14:41:43 net11.bizcooker.com lxd.daemon[698701]: => Starting LXD
Aug 17 14:41:43 net11.bizcooker.com lxd.daemon[698701]: => LXD is ready
Aug 17 14:41:43 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:43 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:44 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:44 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 134.
Aug 17 14:41:44 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:44 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: => Preparing the system
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Loading snap configuration
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Setting up kmod wrapper
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Preparing /boot
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Preparing a clean copy of /run
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Preparing a clean copy of /etc
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Setting up ceph configuration
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Setting up LVM configuration
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Rotating logs
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Setting up ZFS (0.7)
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Escaping the systemd cgroups
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: ==> Escaping the systemd process resource limits
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: => Re-using existing LXCFS
Aug 17 14:41:44 net11.bizcooker.com lxd.daemon[698807]: => Starting LXD
Aug 17 14:41:46 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:46 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:46 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:46 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 135.
Aug 17 14:41:46 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:46 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: => Preparing the system
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Loading snap configuration
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Setting up kmod wrapper
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Preparing /boot
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Preparing a clean copy of /run
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Preparing a clean copy of /etc
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Setting up ceph configuration
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Setting up LVM configuration
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Rotating logs
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Setting up ZFS (0.7)
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Escaping the systemd cgroups
Aug 17 14:41:46 net11.bizcooker.com lxd.daemon[698911]: ==> Escaping the systemd process resource limits
Aug 17 14:41:47 net11.bizcooker.com lxd.daemon[698911]: => Re-using existing LXCFS
Aug 17 14:41:47 net11.bizcooker.com lxd.daemon[698911]: => Starting LXD
Aug 17 14:41:48 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:48 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:49 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:49 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 136.
Aug 17 14:41:49 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:49 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: => Preparing the system
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Loading snap configuration
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Setting up kmod wrapper
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Preparing /boot
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Preparing a clean copy of /run
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Preparing a clean copy of /etc
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Setting up ceph configuration
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Setting up LVM configuration
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Rotating logs
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Setting up ZFS (0.7)
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Escaping the systemd cgroups
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: ==> Escaping the systemd process resource limits
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: => Re-using existing LXCFS
Aug 17 14:41:49 net11.bizcooker.com lxd.daemon[699018]: => Starting LXD
Aug 17 14:41:51 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:51 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:51 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:51 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 137.
Aug 17 14:41:51 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:51 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: => Preparing the system
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Loading snap configuration
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Setting up kmod wrapper
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Preparing /boot
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Preparing a clean copy of /run
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Preparing a clean copy of /etc
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Setting up ceph configuration
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Setting up LVM configuration
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Rotating logs
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Setting up ZFS (0.7)
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Escaping the systemd cgroups
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: ==> Escaping the systemd process resource limits
Aug 17 14:41:51 net11.bizcooker.com lxd.daemon[699122]: => Re-using existing LXCFS
Aug 17 14:41:52 net11.bizcooker.com lxd.daemon[699122]: => Starting LXD
Aug 17 14:41:53 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:53 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:53 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:53 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 138.
Aug 17 14:41:53 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:53 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:53 net11.bizcooker.com lxd.daemon[699228]: => Preparing the system
Aug 17 14:41:53 net11.bizcooker.com lxd.daemon[699228]: ==> Loading snap configuration
Aug 17 14:41:53 net11.bizcooker.com lxd.daemon[699228]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:53 net11.bizcooker.com lxd.daemon[699228]: ==> Setting up kmod wrapper
Aug 17 14:41:53 net11.bizcooker.com lxd.daemon[699228]: ==> Preparing /boot
Aug 17 14:41:53 net11.bizcooker.com lxd.daemon[699228]: ==> Preparing a clean copy of /run
Aug 17 14:41:53 net11.bizcooker.com lxd.daemon[699228]: ==> Preparing a clean copy of /etc
Aug 17 14:41:54 net11.bizcooker.com lxd.daemon[699228]: ==> Setting up ceph configuration
Aug 17 14:41:54 net11.bizcooker.com lxd.daemon[699228]: ==> Setting up LVM configuration
Aug 17 14:41:54 net11.bizcooker.com lxd.daemon[699228]: ==> Rotating logs
Aug 17 14:41:54 net11.bizcooker.com lxd.daemon[699228]: ==> Setting up ZFS (0.7)
Aug 17 14:41:54 net11.bizcooker.com lxd.daemon[699228]: ==> Escaping the systemd cgroups
Aug 17 14:41:54 net11.bizcooker.com lxd.daemon[699228]: ==> Escaping the systemd process resource limits
Aug 17 14:41:54 net11.bizcooker.com lxd.daemon[699228]: => Re-using existing LXCFS
Aug 17 14:41:54 net11.bizcooker.com lxd.daemon[699228]: => Starting LXD
Aug 17 14:41:55 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:55 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:55 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:55 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 139.
Aug 17 14:41:55 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:55 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:55 net11.bizcooker.com lxd.daemon[699344]: => Preparing the system
Aug 17 14:41:55 net11.bizcooker.com lxd.daemon[699344]: ==> Loading snap configuration
Aug 17 14:41:55 net11.bizcooker.com lxd.daemon[699344]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:55 net11.bizcooker.com lxd.daemon[699344]: ==> Setting up kmod wrapper
Aug 17 14:41:55 net11.bizcooker.com lxd.daemon[699344]: ==> Preparing /boot
Aug 17 14:41:55 net11.bizcooker.com lxd.daemon[699344]: ==> Preparing a clean copy of /run
Aug 17 14:41:55 net11.bizcooker.com lxd.daemon[699344]: ==> Preparing a clean copy of /etc
Aug 17 14:41:56 net11.bizcooker.com lxd.daemon[699344]: ==> Setting up ceph configuration
Aug 17 14:41:56 net11.bizcooker.com lxd.daemon[699344]: ==> Setting up LVM configuration
Aug 17 14:41:56 net11.bizcooker.com lxd.daemon[699344]: ==> Rotating logs
Aug 17 14:41:56 net11.bizcooker.com lxd.daemon[699344]: ==> Setting up ZFS (0.7)
Aug 17 14:41:56 net11.bizcooker.com lxd.daemon[699344]: ==> Escaping the systemd cgroups
Aug 17 14:41:56 net11.bizcooker.com lxd.daemon[699344]: ==> Escaping the systemd process resource limits
Aug 17 14:41:56 net11.bizcooker.com lxd.daemon[699344]: => Re-using existing LXCFS
Aug 17 14:41:56 net11.bizcooker.com lxd.daemon[699344]: => Starting LXD
Aug 17 14:41:57 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 14:41:57 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:41:57 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:41:57 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 140.
Aug 17 14:41:57 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:41:57 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: => Preparing the system
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: ==> Loading snap configuration
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: ==> Setting up kmod wrapper
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: ==> Preparing /boot
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: ==> Preparing a clean copy of /run
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: ==> Preparing a clean copy of /etc
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: ==> Setting up ceph configuration
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: ==> Setting up LVM configuration
Aug 17 14:41:57 net11.bizcooker.com lxd.daemon[699446]: ==> Rotating logs
Aug 17 14:41:58 net11.bizcooker.com lxd.daemon[699446]: ==> Setting up ZFS (0.7)
Aug 17 14:41:58 net11.bizcooker.com lxd.daemon[699446]: ==> Escaping the systemd cgroups
Aug 17 14:41:58 net11.bizcooker.com lxd.daemon[699446]: ==> Escaping the systemd process resource limits
Aug 17 14:41:58 net11.bizcooker.com lxd.daemon[699446]: => Re-using existing LXCFS
Aug 17 14:41:58 net11.bizcooker.com lxd.daemon[699446]: => Starting LXD
Aug 17 14:41:58 net11.bizcooker.com lxd.daemon[699446]: lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2018-08-17T14:41:58-0500
Aug 17 14:42:03 net11.bizcooker.com systemd[1]: Reloading Service for snap application lxd.daemon.
Aug 17 14:42:03 net11.bizcooker.com systemd[1]: Reloaded Service for snap application lxd.daemon.
Aug 17 14:42:04 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=137/n/a
Aug 17 14:42:04 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 17 14:42:04 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 17 14:42:04 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 141.
Aug 17 14:42:04 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 14:42:04 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 14:42:04 net11.bizcooker.com lxd.daemon[699757]: => Preparing the system
Aug 17 14:42:04 net11.bizcooker.com lxd.daemon[699757]: ==> Loading snap configuration
Aug 17 14:42:04 net11.bizcooker.com lxd.daemon[699757]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 14:42:04 net11.bizcooker.com lxd.daemon[699757]: ==> Setting up kmod wrapper
Aug 17 14:42:04 net11.bizcooker.com lxd.daemon[699757]: ==> Preparing /boot
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: ==> Preparing a clean copy of /run
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: ==> Preparing a clean copy of /etc
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: ==> Setting up ceph configuration
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: ==> Setting up LVM configuration
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: ==> Rotating logs
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: ==> Setting up ZFS (0.7)
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: ==> Escaping the systemd cgroups
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: ==> Escaping the systemd process resource limits
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: => Re-using existing LXCFS
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: => Starting LXD
Aug 17 14:42:05 net11.bizcooker.com lxd.daemon[699757]: lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2018-08-17T14:42:05-0500
Aug 17 20:36:57 net11.bizcooker.com systemd[1]: Stopping Service for snap application lxd.daemon...
Aug 17 20:36:58 net11.bizcooker.com lxd.daemon[843118]: => Stop reason is: reload
Aug 17 20:36:58 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 17 20:37:06 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: => Preparing the system
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: ==> Loading snap configuration
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: ==> Setting up kmod wrapper
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: ==> Preparing /boot
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: ==> Preparing a clean copy of /run
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: ==> Preparing a clean copy of /etc
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: ==> Setting up ceph configuration
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: ==> Setting up LVM configuration
Aug 17 20:37:06 net11.bizcooker.com lxd.daemon[843363]: ==> Rotating logs
Aug 17 20:37:07 net11.bizcooker.com lxd.daemon[843363]: ==> Setting up ZFS (0.7)
Aug 17 20:37:07 net11.bizcooker.com lxd.daemon[843363]: ==> Escaping the systemd cgroups
Aug 17 20:37:07 net11.bizcooker.com lxd.daemon[843363]: ==> Escaping the systemd process resource limits
Aug 18 08:21:47 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=137/n/a
Aug 18 08:21:47 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Aug 18 08:21:47 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Unit cannot be reloaded because it is inactive.
Aug 18 08:21:47 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Aug 18 08:21:47 net11.bizcooker.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 1.
Aug 18 08:21:47 net11.bizcooker.com systemd[1]: Stopped Service for snap application lxd.daemon.
Aug 18 08:21:47 net11.bizcooker.com systemd[1]: Started Service for snap application lxd.daemon.
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: => Preparing the system
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: ==> Loading snap configuration
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: ==> Setting up mntns symlink (mnt:[4026532292])
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: ==> Setting up kmod wrapper
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: ==> Preparing /boot
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: ==> Preparing a clean copy of /run
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: ==> Preparing a clean copy of /etc
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: ==> Setting up ceph configuration
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: ==> Setting up LVM configuration
Aug 18 08:21:47 net11.bizcooker.com lxd.daemon[118993]: ==> Rotating logs
Aug 18 08:21:48 net11.bizcooker.com lxd.daemon[118993]: ==> Setting up ZFS (0.7)
Aug 18 08:21:48 net11.bizcooker.com lxd.daemon[118993]: ==> Escaping the systemd cgroups
Aug 18 08:21:48 net11.bizcooker.com lxd.daemon[118993]: ==> Escaping the systemd process resource limits
Aug 18 08:21:48 net11.bizcooker.com lxd.daemon[118993]: => Re-using existing LXCFS
Aug 18 08:21:48 net11.bizcooker.com lxd.daemon[118993]: => Starting LXD
Aug 18 08:21:48 net11.bizcooker.com lxd.daemon[118993]: lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2018-08-18T08:21:48-0500
davidfavor commented 6 years ago

Clarification, I'm seeing this problem occur on LXD-3.4.8393 machines repeatedly, so the steps above are required to be repeated at regular intervals to clear the problem.

davidfavor commented 6 years ago

Problem is intermittent. Reoccurs repeatedly on some machines. Other machines appear to clear the problem after going through the fix once.

davidfavor commented 6 years ago

This also fixes problem...

net11 # systemctl restart snap.lxd.daemon
davidfavor commented 6 years ago

Problem seems to be getting worse. Happening every few minutes now.

Had to create a background job to test for "500 Internal Server Error" + do the restart trick to vix problem.

Simple PERL script to keep problem fixed.

my $restart = "systemctl restart snap.lxd.daemon";

while (1) {

   my $res = system("lxc info >/dev/null 2>&1");

   if ($res) {
      my ($date) = `date`; chop $date;
      print "# restart: $date: $restart\n";
      system($restart);
   }   

   sleep(10);

}
stgraber commented 6 years ago

@davidfavor do you actually see multiple copies of the lxd process when this happens?

I wonder if you're not instead running into a database problem.

Can you look in /var/snap/lxd/common/lxd/logs/lxd.log for database errors.

davidfavor commented 6 years ago

I'll check on this next time the problem reoccurs... which... sigh...

Of course, since I wrote a script to auto-fix the problem + print out a time stamp anytime problem occurs...

The problem has not reoccurred.

So problem occurred on several machines early this morning + hasn't reoccurred since.

/var/snap/lxd/common/lxd/logs/lxd.log seems good. No oddball errors, just normal diagnostics when containers restart.

If the problem reoccurs, I'll update this ticket.

stgraber commented 6 years ago

@davidfavor ok, hopefully it's the same issue as #4947 and we can figure it out soon

davidfavor commented 6 years ago

Seems to have cleared up now.

Go ahead + close this out.

Thanks!

davidfavor commented 6 years ago

What I'm running which has seemed to fix problem...

net14 # snap list
Name  Version  Rev   Tracking  Publisher   Notes
core  16-2.35  5328  stable    canonical*  core
lxd   3.4      8415  stable    canonical*  -