search

subutai-io / agent

Subutai Agent is a tool that provides CLI to control Subutai infrastructure, and is a daemon that receives and performs Management commands through secured channels
https://subutai.io
15 stars 15 forks source link

Mountpoint `/var/lib/lxc` getting lost randomly #804

Closed jadilet closed 5 years ago

jadilet commented 6 years ago

I have created master peer via vagrant and libvirt(KVM) and registered to Bazaar.

subutai@subutai:~$ sudo subutai -v
8.1.0+201809210319160
subutai@subutai:~$

While building environment on Bazaar, I got this:

ERROR - Building containers. [Task (Clone sds-T5D from debian-stretch) 
succeeded on host subutai [00:00:02], 
Task (Clone ansible-server-395-tGp from generic-ansible) 
failed on host subutai [Error cloning container ansible-server-395-tGp-qxi-2-3: 
Failed to clone container: time="2018-09-25 03:32:46" 
level=fatal msg="Lxc directory /var/lib/lxc/ not mounted" , 
result FAILED]]
subutai@subutai:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=2013660k,nr_inodes=503415,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=405076k,mode=755)
/dev/mapper/main-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/main-usr on /usr type ext4 (rw,relatime,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10519)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/mapper/main-var on /var type ext4 (rw,relatime,data=ordered)
/dev/mapper/main-home on /home type ext4 (rw,relatime,data=ordered)
/dev/mapper/main-opt on /opt type ext4 (rw,relatime,data=ordered)
/dev/vda1 on /boot type ext3 (rw,relatime,data=ordered)
subutai on /subutai type zfs (rw,xattr,noacl)
subutai/fs/debian-stretch:subutai:0.4.5 on /var/lib/lxc/debian-stretch:subutai:0.4.5 type zfs (rw,xattr,noacl)
subutai/fs/debian-stretch:subutai:0.4.5/home on /var/lib/lxc/debian-stretch:subutai:0.4.5/home type zfs (ro,xattr,noacl)
subutai/fs/debian-stretch:subutai:0.4.5/opt on /var/lib/lxc/debian-stretch:subutai:0.4.5/opt type zfs (ro,xattr,noacl)
subutai/fs/debian-stretch:subutai:0.4.5/rootfs on /var/lib/lxc/debian-stretch:subutai:0.4.5/rootfs type zfs (ro,xattr,noacl)
subutai/fs/debian-stretch:subutai:0.4.5/var on /var/lib/lxc/debian-stretch:subutai:0.4.5/var type zfs (ro,xattr,noacl)
subutai/fs/management on /var/lib/lxc/management type zfs (rw,xattr,noacl)
subutai/fs/management/home on /var/lib/lxc/management/home type zfs (rw,xattr,noacl)
subutai/fs/management/opt on /var/lib/lxc/management/opt type zfs (rw,xattr,noacl)
subutai/fs/management/rootfs on /var/lib/lxc/management/rootfs type zfs (rw,xattr,noacl)
subutai/fs/management/var on /var/lib/lxc/management/var type zfs (rw,xattr,noacl)
subutai/fs/management:jenkins:7.0.8 on /var/lib/lxc/management:jenkins:7.0.8 type zfs (rw,xattr,noacl)
subutai/fs/management:jenkins:7.0.8/home on /var/lib/lxc/management:jenkins:7.0.8/home type zfs (ro,xattr,noacl)
subutai/fs/management:jenkins:7.0.8/opt on /var/lib/lxc/management:jenkins:7.0.8/opt type zfs (ro,xattr,noacl)
subutai/fs/management:jenkins:7.0.8/rootfs on /var/lib/lxc/management:jenkins:7.0.8/rootfs type zfs (ro,xattr,noacl)
subutai/fs/management:jenkins:7.0.8/var on /var/lib/lxc/management:jenkins:7.0.8/var type zfs (ro,xattr,noacl)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=405072k,mode=700,uid=1000,gid=1000)
subutai/fs/fddddd-OXa-vtm-1-4 on /var/lib/lxc/fddddd-OXa-vtm-1-4 type zfs (rw,xattr,noacl)
subutai/fs/fddddd-OXa-vtm-1-4/rootfs on /var/lib/lxc/fddddd-OXa-vtm-1-4/rootfs type zfs (rw,xattr,noacl)
subutai/fs/fddddd-OXa-vtm-1-4/home on /var/lib/lxc/fddddd-OXa-vtm-1-4/home type zfs (rw,xattr,noacl)
subutai/fs/fddddd-OXa-vtm-1-4/var on /var/lib/lxc/fddddd-OXa-vtm-1-4/var type zfs (rw,xattr,noacl)
subutai/fs/fddddd-OXa-vtm-1-4/opt on /var/lib/lxc/fddddd-OXa-vtm-1-4/opt type zfs (rw,xattr,noacl)
subutai/fs/sds-T5D-pgy-2-4 on /var/lib/lxc/sds-T5D-pgy-2-4 type zfs (rw,xattr,noacl)
subutai/fs/sds-T5D-pgy-2-4/rootfs on /var/lib/lxc/sds-T5D-pgy-2-4/rootfs type zfs (rw,xattr,noacl)
subutai/fs/sds-T5D-pgy-2-4/home on /var/lib/lxc/sds-T5D-pgy-2-4/home type zfs (rw,xattr,noacl)
subutai/fs/sds-T5D-pgy-2-4/var on /var/lib/lxc/sds-T5D-pgy-2-4/var type zfs (rw,xattr,noacl)
subutai/fs/sds-T5D-pgy-2-4/opt on /var/lib/lxc/sds-T5D-pgy-2-4/opt type zfs (rw,xattr,noacl)
subutai@subutai:~$ 
subutai@subutai:~$ df
Filesystem                                     1K-blocks   Used Available Use% Mounted on
udev                                             2013660      0   2013660   0% /dev
tmpfs                                             405076  37832    367244  10% /run
/dev/mapper/main-root                            1934672 466488   1351860  26% /
/dev/mapper/main-usr                             7808392 799840   6592196  11% /usr
tmpfs                                            2025368      0   2025368   0% /dev/shm
tmpfs                                               5120      0      5120   0% /run/lock
tmpfs                                            2025368      0   2025368   0% /sys/fs/cgroup
/dev/mapper/main-var                             7808392 489028   6903008   7% /var
/dev/mapper/main-home                            1934672   5904   1812444   1% /home
/dev/mapper/main-opt                             3869352  15608   3637476   1% /opt
/dev/vda1                                         967320  63176    854176   7% /boot
subutai                                        103513088      0 103513088   0% /subutai
subutai/fs/debian-stretch:subutai:0.4.5        103513088      0 103513088   0% /var/lib/lxc/debian-stretch:subutai:0.4.5
subutai/fs/debian-stretch:subutai:0.4.5/home   103513088      0 103513088   0% /var/lib/lxc/debian-stretch:subutai:0.4.5/home
subutai/fs/debian-stretch:subutai:0.4.5/opt    103513088      0 103513088   0% /var/lib/lxc/debian-stretch:subutai:0.4.5/opt
subutai/fs/debian-stretch:subutai:0.4.5/rootfs 103762560 249472 103513088   1% /var/lib/lxc/debian-stretch:subutai:0.4.5/rootfs
subutai/fs/debian-stretch:subutai:0.4.5/var    103521024   7936 103513088   1% /var/lib/lxc/debian-stretch:subutai:0.4.5/var
subutai/fs/management                          103513216    128 103513088   1% /var/lib/lxc/management
subutai/fs/management/home                     103513088      0 103513088   0% /var/lib/lxc/management/home
subutai/fs/management/opt                      103722752 209664 103513088   1% /var/lib/lxc/management/opt
subutai/fs/management/rootfs                   104242304 729216 103513088   1% /var/lib/lxc/management/rootfs
subutai/fs/management/var                      103669120 156032 103513088   1% /var/lib/lxc/management/var
subutai/fs/management:jenkins:7.0.8            103513088      0 103513088   0% /var/lib/lxc/management:jenkins:7.0.8
subutai/fs/management:jenkins:7.0.8/home       103513088      0 103513088   0% /var/lib/lxc/management:jenkins:7.0.8/home
subutai/fs/management:jenkins:7.0.8/opt        103637888 124800 103513088   1% /var/lib/lxc/management:jenkins:7.0.8/opt
subutai/fs/management:jenkins:7.0.8/rootfs     104242304 729216 103513088   1% /var/lib/lxc/management:jenkins:7.0.8/rootfs
subutai/fs/management:jenkins:7.0.8/var        103597440  84352 103513088   1% /var/lib/lxc/management:jenkins:7.0.8/var
tmpfs                                             405072      0    405072   0% /run/user/1000
subutai/fs/fddddd-OXa-vtm-1-4                  103513088      0 103513088   0% /var/lib/lxc/fddddd-OXa-vtm-1-4
subutai/fs/fddddd-OXa-vtm-1-4/rootfs           103762560 249472 103513088   1% /var/lib/lxc/fddddd-OXa-vtm-1-4/rootfs
subutai/fs/fddddd-OXa-vtm-1-4/home             103513088      0 103513088   0% /var/lib/lxc/fddddd-OXa-vtm-1-4/home
subutai/fs/fddddd-OXa-vtm-1-4/var              103521024   7936 103513088   1% /var/lib/lxc/fddddd-OXa-vtm-1-4/var
subutai/fs/fddddd-OXa-vtm-1-4/opt              103513088      0 103513088   0% /var/lib/lxc/fddddd-OXa-vtm-1-4/opt
subutai/fs/sds-T5D-pgy-2-4                     103513088      0 103513088   0% /var/lib/lxc/sds-T5D-pgy-2-4
subutai/fs/sds-T5D-pgy-2-4/rootfs              103762560 249472 103513088   1% /var/lib/lxc/sds-T5D-pgy-2-4/rootfs
subutai/fs/sds-T5D-pgy-2-4/home                103513088      0 103513088   0% /var/lib/lxc/sds-T5D-pgy-2-4/home
subutai/fs/sds-T5D-pgy-2-4/var                 103521024   7936 103513088   1% /var/lib/lxc/sds-T5D-pgy-2-4/var
subutai/fs/sds-T5D-pgy-2-4/opt                 103513088      0 103513088   0% /var/lib/lxc/sds-T5D-pgy-2-4/opt
subutai@subutai:~$ 
subutai@subutai:~$ mountpoint /var/lib/lxc/
/var/lib/lxc/ is not a mountpoint
subutai@subutai:~$
dilshat commented 6 years ago

@lbthomsen seems subutai/fs -> /var/lib/lxc mountpoint gets lost during peer regular operation and can not be restored without destroying all installed templates and containers

Almansherov commented 6 years ago

Faced an error on masterbazaar while using qavms-kg3-mast peer for building Lamp Blueprint.

ERROR - Building containers. [Task (Clone dokuwiki-KJb from debian-stretch) succeeded on host subutai [00:00:15], Task (Clone ansible-server-237-9Bb from generic-ansible) failed on host subutai [Error cloning container ansible-server-237-9Bb-5md-2-3: Failed to clone container: time="2018-09-28 11:00:23" level=fatal msg="Lxc directory /var/lib/lxc/ not mounted" , result FAILED]]

etursunbaev commented 6 years ago

Weird it fails on import commands

root@subutai:~# subutai -d import generic-ansible
DEBU[2018-09-28 11:29:43] Checking mountpoint /var/lib/lxc/ is not a mountpoint
, exit status 1 
FATA[2018-09-28 11:29:43] Lxc directory /var/lib/lxc/ not mounted      
root@subutai:~# subutai -d import oraclejdk8
DEBU[2018-09-28 11:30:41] Checking mountpoint /var/lib/lxc/ is not a mountpoint
, exit status 1 
FATA[2018-09-28 11:30:41] Lxc directory /var/lib/lxc/ not mounted      
root@subutai:~# 
root@subutai:~#

Creating containers from clone works fine

dilshat commented 6 years ago

Yes seems the check using mountpoint command should be revisited

etursunbaev commented 5 years ago

I also suspect that subutai/stretch-master box might be borked I faced this issue with this newly box

dilshat commented 5 years ago

@jadilet ^

dilshat commented 5 years ago

@etursunbaev btw I changed the way of checking mountpoint. Let me know if you face any issues on master/dev branches with fresh peers