search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
4.02k stars 3.49k forks source link

lxc-start: The container failed to start (21.02) #15433

Closed erdoukki closed 3 years ago

erdoukki commented 3 years ago

https://forum.openwrt.org/t/lxc-mvebu-snapshot-21-02/94033

I have just installed the snapshot 21-02 for MVEBU EspressoBIN... I am surprised that LXC looks like to be integrated in this future release (great news if it is) ! I am trying to make a LXC container but get an error : 

root@OWRT-DEV:/# lxc-create --name myBUS --template download -- --dist debian --
release buster --arch arm64                                                     
Failed to create lock for myBUS                                                 
lxc-create: myBUS: tools/lxc_create.c: main: 260 Failed to create lxc container 
root@OWRT-DEV:/# uname -ar                                                      
Linux OWRT-DEV 5.4.111 #0 SMP Wed Apr 14 06:42:38 2021 aarch64 GNU/Linux        
root@OWRT-DEV:/# lxc-checkconfig                                                
LXC version 4.0.5                                                               
--- Namespaces ---                                                              
Namespaces: enabled                                                             
Utsname namespace: enabled                                                      
Ipc namespace: enabled                                                          
Pid namespace: enabled                                                          
User namespace: enabled                                                         
Network namespace: enabled                                                      

--- Control groups ---                                                          
Cgroups: enabled                                                                

Cgroup v1 mount points:                                                         
/sys/fs/cgroup/cpuset                                                           
/sys/fs/cgroup/cpu                                                              
/sys/fs/cgroup/cpuacct                                                          
/sys/fs/cgroup/blkio                                                            
/sys/fs/cgroup/memory                                                           
/sys/fs/cgroup/pids                                                             
/sys/fs/cgroup/rdma                                                             

Cgroup v2 mount points:                                                         

Cgroup v1 systemd controller: missing                                           
Cgroup v1 freezer controller: missing                                           
Cgroup v1 clone_children flag: enabled                                          
Cgroup device: missing                                                          
Cgroup sched: enabled                                                           
Cgroup cpu account: enabled                                                     
Cgroup memory controller: enabled                                               
Cgroup cpuset: enabled                                                          

--- Misc ---                                                                    
Veth pair device: enabled, loaded                                               
Macvlan: enabled, not loaded                                                    
Vlan: enabled, not loaded                                                       
Bridges: enabled, not loaded                                                    
Advanced netfilter: enabled, not loaded                                         
CONFIG_NF_NAT_IPV4: missing                                                     
CONFIG_NF_NAT_IPV6: missing                                                     
CONFIG_IP_NF_TARGET_MASQUERADE: missing                                         
CONFIG_IP6_NF_TARGET_MASQUERADE: missing                                        
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded                        
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded                              
FUSE (for use with lxcfs): enabled, not loaded                                  

--- Checkpoint/Restore ---                                                      
checkpoint restore: missing                                                     
CONFIG_FHANDLE: enabled                                                         
CONFIG_EVENTFD: enabled                                                         
CONFIG_EPOLL: enabled                                                           
CONFIG_UNIX_DIAG: missing                                                       
CONFIG_INET_DIAG: missing                                                       
CONFIG_PACKET_DIAG: missing                                                     
CONFIG_NETLINK_DIAG: enabled                                                    
File capabilities:                                                              

Note : Before booting a new kernel, you can check its configuration             
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig                         

root@OWRT-DEV:/# 

root@OWRT-DEV:~# ls /sys/fs/cgroup/
blkio    cpu      cpuacct  cpuset   memory   pids     rdma
service lxc-auto enable
service lxc-auto boot

then... 

root@OWRT-DEV:~# ls /sys/fs/cgroup/
blkio    cpu      cpuacct  cpuset   memory   pids     rdma     systemd
root@OWRT-DEV:~# lxc-checkconfig 
LXC version 4.0.5
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points: 
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/cpu
/sys/fs/cgroup/cpuacct
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/pids
/sys/fs/cgroup/rdma
/sys/fs/cgroup/systemd

Cgroup v2 mount points: 

Cgroup v1 freezer controller: missing
Cgroup v1 clone_children flag: enabled
Cgroup device: missing
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: missing
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: missing
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: enabled
File capabilities: 

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

root@OWRT-DEV:/# lxc-create --name myBUSTER --template download -- --dist debian
 --release buster --arch arm64 --no-validate                                    
Downloading the image index                                                     
WARNING: Running without gpg validation!                                        
Downloading the rootfs                                                          
Downloading the metadata                                                        
The image cache is now ready                                                    
Unpacking the rootfs                                                            

---                                                                             
You just created a Debian buster arm64 (20210415_05:24) container.              

To enable SSH, run: apt install openssh-server                                  
No default root or user password are set by LXC.

still an issue : 

root@OWRT-DEV:~# lxc-start -n myBUSTER --foreground -l TRACE
lxc-start: myBUSTER: cgroups/cgfsng.c: cg_legacy_set_data: 2824 No such file or directory - Failed to setup limits for the "devices" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy
   lxc-start: myBUSTER: cgroups/cgfsng.c: cgfsng_setup_limits_legacy: 2873 No such file or directory - Failed to set "devices.deny" to "a"
                                                          lxc-start: myBUSTER: start.c: lxc_spawn: 1828 Failed to setup legacy device cgroup controller limits
                                                                              lxc-start: myBUSTER: start.c: __lxc_start: 1999 Failed to spawn container "myBUSTER"
  lxc-start: myBUSTER: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: myBUSTER: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options


root@OWRT-DEV:~# cat /proc/cgroups 
#subsys_name    hierarchy   num_cgroups enabled
cpuset  1   2   1
cpu 2   2   1
cpuacct 3   2   1
blkio   4   2   1
memory  5   8   1
pids    6   2   1
rdma    7   2   1

is devices missing ??? why ?

erdoukki commented 3 years ago

looks really close to https://github.com/openwrt/packages/issues/13037

erdoukki commented 3 years ago 
root@OWRT-DEV:~# cat /srv/lxc/myBUSTER/config 
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist debian --release buster --arch arm64 --no-validate
# Template script checksum (SHA-1): 26e72660447e5905798fa16f5a022191b590f8fc
# For additional config options, please look at lxc.container.conf(5)

##TWEAK
##lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = linux64

# Container specific configuration
lxc.rootfs.path = dir:/srv/lxc/myBUSTER/rootfs
lxc.uts.name = myBUSTER

# Network configuration
lxc.net.0.type = empty
dangowrt commented 3 years ago

Apparently LXC still tries to use cgroupv1 subsystems. OpenWrt only supports cgroupv2, support for (legacy) cgroupv1 is neither planned nor intended. Best would probably be to update LXC to use cgroupv2 instead (ie. eBPF for devices).

erdoukki commented 3 years ago

yes, and it can be fixed by modifying the /usr/share/lxc/config/common.conf like this : 

root@OWRT-DEV:~# diff /usr/share/lxc/config/common.conf.orig /usr/share/lxc/conf
ig/common.conf --unified
--- /usr/share/lxc/config/common.conf.orig  2021-04-15 14:50:15.703804432 +0200
+++ /usr/share/lxc/config/common.conf   2021-04-15 14:50:45.414118083 +0200
@@ -18,31 +18,31 @@
 # Default legacy cgroup configuration
 #
 # CGroup allowlist
-lxc.cgroup.devices.deny = a
+#lxc.cgroup.devices.deny = a
 ## Allow any mknod (but not reading/writing the node)
-lxc.cgroup.devices.allow = c *:* m
-lxc.cgroup.devices.allow = b *:* m
+#lxc.cgroup.devices.allow = c *:* m
+#lxc.cgroup.devices.allow = b *:* m
 ## Allow specific devices
 ### /dev/null
-lxc.cgroup.devices.allow = c 1:3 rwm
+#lxc.cgroup.devices.allow = c 1:3 rwm
 ### /dev/zero
-lxc.cgroup.devices.allow = c 1:5 rwm
+#lxc.cgroup.devices.allow = c 1:5 rwm
 ### /dev/full
-lxc.cgroup.devices.allow = c 1:7 rwm
+#lxc.cgroup.devices.allow = c 1:7 rwm
 ### /dev/tty
-lxc.cgroup.devices.allow = c 5:0 rwm
+#lxc.cgroup.devices.allow = c 5:0 rwm
 ### /dev/console
-lxc.cgroup.devices.allow = c 5:1 rwm
+#lxc.cgroup.devices.allow = c 5:1 rwm
 ### /dev/ptmx
-lxc.cgroup.devices.allow = c 5:2 rwm
+#lxc.cgroup.devices.allow = c 5:2 rwm
 ### /dev/random
-lxc.cgroup.devices.allow = c 1:8 rwm
+#lxc.cgroup.devices.allow = c 1:8 rwm
 ### /dev/urandom
-lxc.cgroup.devices.allow = c 1:9 rwm
+#lxc.cgroup.devices.allow = c 1:9 rwm
 ### /dev/pts/*
-lxc.cgroup.devices.allow = c 136:* rwm
+#lxc.cgroup.devices.allow = c 136:* rwm
 ### fuse
-lxc.cgroup.devices.allow = c 10:229 rwm
+#lxc.cgroup.devices.allow = c 10:229 rwm

 # Default unified cgroup configuration
 #
root@OWRT-DEV:~#

then lxc-start works correctly...


root@OWRT-DEV:/# lxc-start -n myBUSTER --foreground --logpriority TRACE         
systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK 
+SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +EL
FUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)                        
Detected virtualization lxc.                                                    
Detected architecture arm64.                                                    

Welcome to Debian GNU/Linux 10 (buster)!                                        

Set hostname to <myBUSTER>.                                                     
Couldn't move remaining userspace processes, ignoring: Input/output error       
[  OK  ] Created slice system-container\x2dgetty.slice.                         
[  OK  ] Started Forward Password Requests to Wall Directory Watch.             
[  OK  ] Listening on Journal Socket.                                           
[  OK  ] Listening on Journal Socket (/dev/log).                                
         Starting Apply Kernel Variables...                                     
[  OK  ] Reached target Swap.                                                   
         Starting Helper to synchronize boot up for ifupdown...                 
[  OK  ] Listening on initctl Compatibility Named Pipe.                         
[  OK  ] Reached target Remote File Systems.                                    
         Mounting POSIX Message Queue File System...                            
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.         
[  OK  ] Reached target Local Encrypted Volumes.                                
[  OK  ] Reached target Paths.                                                  
         Starting Journal Service...                                            
[  OK  ] Created slice User and Session Slice.                                  
[  OK  ] Reached target Slices.                                                 
         Starting Remount Root and Kernel File Systems...                       
[  OK  ] Created slice system-getty.slice.                                      
[  OK  ] Mounted POSIX Message Queue File System.                               
[  OK  ] Started Helper to synchronize boot up for ifupdown.                    
[  OK  ] Started Apply Kernel Variables.                                        
[  OK  ] Started Remount Root and Kernel File Systems.                          
         Starting Create System Users...                                        
[  OK  ] Started Journal Service.                                               
         Starting Flush Journal to Persistent Storage...                        
[  OK  ] Started Create System Users.                                           
         Starting Create Static Device Nodes in /dev...                         
[  OK  ] Started Flush Journal to Persistent Storage.                           
[  OK  ] Started Create Static Device Nodes in /dev.                            
[  OK  ] Reached target Local File Systems (Pre).                               
[  OK  ] Reached target Local File Systems.                                     
         Starting Raise network interfaces...                                   
         Starting Create Volatile Files and Directories...                      
[  OK  ] Started Create Volatile Files and Directories.                         
         Starting Update UTMP about System Boot/Shutdown...                     
[  OK  ] Reached target System Time Synchronized.                               
[  OK  ] Started Update UTMP about System Boot/Shutdown.                        
[  OK  ] Reached target System Initialization.                                  
[  OK  ] Listening on D-Bus System Message Bus Socket.                          
[  OK  ] Reached target Sockets.                                                
[  OK  ] Reached target Basic System.                                           
         Starting Login Service...                                              
[  OK  ] Started Daily apt download activities.                                 
[  OK  ] Started D-Bus System Message Bus.                                      
[  OK  ] Started Daily apt upgrade and clean activities.                        
[  OK  ] Started Daily Cleanup of Temporary Directories.                        
[  OK  ] Reached target Timers.                                                 
[FAILED] Failed to start Raise network interfaces.                              
See 'systemctl status networking.service' for details.                          
[  OK  ] Reached target Network.                                                
         Starting Permit User Sessions...                                       
[  OK  ] Started Login Service.                                                 
[  OK  ] Started Permit User Sessions.                                          
[  OK  ] Started Console Getty.                                                 
[  OK  ] Started Container Getty on /dev/pts/0.                                 
[  OK  ] Started Container Getty on /dev/pts/2.                                 
[  OK  ] Started Container Getty on /dev/pts/3.                                 
[  OK  ] Started Container Getty on /dev/pts/1.                                 
[  OK  ] Reached target Login Prompts.                                          
[  OK  ] Reached target Multi-User System.                                      
[  OK  ] Reached target Graphical Interface.                                    
         Starting Update UTMP about System Runlevel Changes...                  
[  OK  ] Started Update UTMP about System Runlevel Changes.                     

Debian GNU/Linux 10 myBUSTER console                                            

myBUSTER login: