search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 931 forks source link

"Error: EOF" after exec command on container. CGROUP errors #7104

Closed 4ePTuk closed 4 years ago

4ePTuk commented 4 years ago

Required information

` config: {} api_extensions:

Executing lxc exec Container bash returns "EOF" error. In container log there are cgoup errors. This happens on every container i have. This started after UPS failed to powerup server and it shutted down

Steps to reproduce

  1. lxc init images:debian/9/amd64 Blog
  2. lxc exec Blog bash

Information to attach

Name: Blog Location: none Remote: unix:// Architecture: x86_64 Created: 2020/03/28 16:48 UTC Status: Running Type: container Profiles: default Pid: 2826 Ips: eth0: inet 192.168.31.120 veth6bed81f8 eth0: inet6 fe80::216:3eff:fe0e:98a veth6bed81f8 lo: inet 127.0.0.1 lo: inet6 ::1 Resources: Processes: 6 CPU usage: CPU usage (in seconds): 0 Memory usage: Memory (current): 14.45MB Network usage: eth0: Bytes received: 2.58MB Bytes sent: 1.99kB Packets received: 8619 Packets sent: 21 lo: Bytes received: 0B Bytes sent: 0B Packets received: 0 Packets sent: 0

Log:

lxc Blog 20200330091342.274 WARN cgfsng - cgroups/cgfsng.c:cg_unified_delegate:2906 - No such file or directory - Failed to read /sys/kernel/cgroup/delegate lxc Blog 20200330091342.291 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1136 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.Blog" lxc Blog 20200330091342.312 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1136 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.Blog" lxc Blog 20200330091342.334 ERROR utils - utils.c:lxc_can_use_pidfd:1834 - Kernel does not support pidfds lxc Blog 20200330091342.375 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1448 - No such file or directory - Failed to fchownat(17, cgroup.threads, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW ) lxc Blog 20200330091909.468 WARN cgfsng - cgroups/cgfsng.c:cg_unified_delegate:2906 - No such file or directory - Failed to read /sys/kernel/cgroup/delegate lxc Blog 20200330091909.471 ERROR cgfsng - cgroups/cgfsng.c:cgroup_attach_leaf:2081 - Permission denied - Failed to attach to unified cgroup lxc Blog 20200330091909.471 ERROR conf - conf.c:userns_exec_minimal:4194 - Permission denied - Running function in new user namespace failed

brauner commented 4 years ago

Is LXD installed via snap?

4ePTuk commented 4 years ago

yes I don't like those cgoup errors...

brauner commented 4 years ago

Are you running the edge snap? It should already be fixed in there if it's the issue I'm suspecting.

4ePTuk commented 4 years ago

Here is my version snap --version snap 2.43.3 snapd 2.43.3 series 16 debian 9 kernel 4.9.0-11-amd64

I think, that i should update it to latest...

brauner commented 4 years ago

On Mon, Mar 30, 2020 at 03:28:12AM -0700, Dmitry wrote:

Here is my version snap --version snap 2.43.3 snapd 2.43.3 series 16 debian 9 kernel 4.9.0-11-amd64

Sorry, I meant if you switch to the edge channel for the LXD snap

snap install --channel=edge

or

snap refresh --channel=edge

Though only do this if you're not running anything important rn.

Christian

4ePTuk commented 4 years ago 
lxd --version
3.23

It's already stable channel stable: 3.23 2020-03-28 (14066) 70MB -

brauner commented 4 years ago

Can you paste the output of:

snap info lxd

please

4ePTuk commented 4 years ago 
name:      lxd
summary:   System container manager and API
publisher: Canonical✓
store-url: https://snapcraft.io/lxd
contact:   https://github.com/lxc/lxd/issues
license:   unset
description: |
  **LXD is a system container manager**

  With LXD you can run hundreds of containers of a variety of Linux
  distributions, apply resource limits, pass in directories, USB devices
  or GPUs and setup any network and storage you want.

  LXD containers are lightweight, secure by default and a great
  alternative to running Linux virtual machines.

  **Run any Linux distribution you want**

  Pre-made images are available for Ubuntu, Alpine Linux, ArchLinux,
  CentOS, Debian, Fedora, Gentoo, OpenSUSE and more.

  A full list of available images can be found here: https://images.linuxcontainers.org

  Can't find the distribution you want? It's easy to make your own images too, either using our
  `distrobuilder` tool or by assembling your own image tarball by hand.

  **Containers at scale**

  LXD is network aware and all interactions go through a simple REST API,
  making it possible to remotely interact with containers on remote
  systems, copying and moving them as you wish.

  Want to go big? LXD also has built-in clustering support,
  letting you turn dozens of servers into one big LXD server.

  **Configuration options**

  Supported options for the LXD snap (`snap set lxd KEY=VALUE`):
   - criu.enable: Enable experimental live-migration support [default=false]
   - daemon.debug: Increases logging to debug level [default=false]
   - daemon.group: Group of users that can interact with LXD [default=lxd]
   - ceph.builtin: Use snap-specific ceph configuration [default=false]
   - openvswitch.builtin: Run a snap-specific OVS daemon [default=false]

  Documentation: https://lxd.readthedocs.io
commands:
  - lxd.benchmark
  - lxd.buginfo
  - lxd.check-kernel
  - lxd.lxc
  - lxd
  - lxd.migrate
services:
  lxd.activate: oneshot, enabled, inactive
  lxd.daemon:   simple, enabled, active
snap-id:      J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking:     latest/stable
refresh-date: 2 days ago, at 07:17 MSK
channels:
  stable:         3.23        2020-03-28 (14066) 70MB -
  candidate:      3.23        2020-03-28 (14095) 70MB -
  beta:           3.23        2020-03-28 (14096) 60MB -
  edge:           git-350df50 2020-03-29 (14114) 60MB -
  3.23/stable:    3.23        2020-03-28 (14066) 70MB -
  3.23/candidate: 3.23        2020-03-28 (14095) 70MB -
  3.23/beta:      ↑
  3.23/edge:      ↑
  3.22/stable:    3.22        2020-03-18 (13901) 70MB -
  3.22/candidate: 3.22        2020-03-19 (13911) 70MB -
  3.22/beta:      ↑
  3.22/edge:      ↑
  3.21/stable:    3.21        2020-02-24 (13522) 69MB -
  3.21/candidate: 3.21        2020-03-04 (13588) 69MB -
  3.21/beta:      ↑
  3.21/edge:      ↑
  3.20/stable:    3.20        2020-02-06 (13300) 69MB -
  3.20/candidate: 3.20        2020-02-06 (13300) 69MB -
  3.20/beta:      ↑
  3.20/edge:      ↑
  3.19/stable:    3.19        2020-01-27 (13162) 67MB -
  3.19/candidate: 3.19        2020-01-27 (13162) 67MB -
  3.19/beta:      ↑
  3.19/edge:      ↑
  3.18/stable:    3.18        2019-12-02 (12631) 57MB -
  3.18/candidate: 3.18        2019-12-02 (12631) 57MB -
  3.18/beta:      ↑
  3.18/edge:      ↑
  3.0/stable:     3.0.4       2019-10-10 (11348) 55MB -
  3.0/candidate:  3.0.4       2019-10-10 (11348) 55MB -
  3.0/beta:       ↑
  3.0/edge:       git-81b81b9 2019-10-10 (11362) 55MB -
  2.0/stable:     2.0.11      2019-10-10  (8023) 28MB -
  2.0/candidate:  2.0.11      2019-10-10  (8023) 28MB -
  2.0/beta:       ↑
  2.0/edge:       git-160221d 2020-01-13 (12854) 27MB -
installed:        3.23                   (14066) 70MB -
brauner commented 4 years ago

Ok, so can you - if your workload does allow it - try with:

snap refresh --channel=latest/candidate

and then report back if you still have the issue?

brauner commented 4 years ago

(Daemon might need to be restarted after that with systemctl restart snap.lxd.daemon.service.

4ePTuk commented 4 years ago

ok, I made snap refresh lxd --channel=latest/candidate restarted daemon and rebooted server...still EOF error (

services:
  lxd.activate: oneshot, enabled, inactive
  lxd.daemon:   simple, enabled, active
snap-id:      J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking:     latest/candidate
refresh-date: today at 14:00 MSK
lxc exec Blog bash
Error: EOF
lxc info Blog --show-log
Name: Blog
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/03/28 16:48 UTC
Status: Running
Type: container
Profiles: default
Pid: 1396
Ips:
  eth0: inet    192.168.31.120  veth4528aad7
  eth0: inet6   fe80::216:3eff:fe0e:98a veth4528aad7
  lo:   inet    127.0.0.1
  lo:   inet6   ::1
Resources:
  Processes: 6
  CPU usage:
    CPU usage (in seconds): 0
  Memory usage:
    Memory (current): 32.15MB
  Network usage:
    eth0:
      Bytes received: 257.71kB
      Bytes sent: 1.70kB
      Packets received: 1044
      Packets sent: 16
    lo:
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0

Log:

lxc Blog 20200330110329.915 WARN     cgfsng - cgroups/cgfsng.c:cg_unified_delegate:2915 - No such file or directory - Failed to read /sys/kernel/cgroup/delegate
lxc Blog 20200330110329.918 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1142 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.Blog"
lxc Blog 20200330110329.920 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1142 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.Blog"
lxc Blog 20200330110329.922 ERROR    utils - utils.c:lxc_can_use_pidfd:1834 - Kernel does not support pidfds
lxc Blog 20200330110329.929 WARN     cgfsng - cgroups/cgfsng.c:fchowmodat:1454 - No such file or directory - Failed to fchownat(17, cgroup.threads, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc Blog 20200330110358.846 WARN     cgfsng - cgroups/cgfsng.c:cg_unified_delegate:2915 - No such file or directory - Failed to read /sys/kernel/cgroup/delegate
lxc Blog 20200330110358.848 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_attach_leaf:2087 - Permission denied - Failed to attach to unified cgroup
lxc Blog 20200330110358.848 ERROR    conf - conf.c:userns_exec_minimal:4194 - Permission denied - Running function in new user namespace failed
lxc Blog 20200330110525.413 WARN     cgfsng - cgroups/cgfsng.c:cg_unified_delegate:2915 - No such file or directory - Failed to read /sys/kernel/cgroup/delegate
lxc Blog 20200330110525.415 ERROR    cgfsng - cgroups/cgfsng.c:cgroup_attach_leaf:2087 - Permission denied - Failed to attach to unified cgroup
lxc Blog 20200330110525.415 ERROR    conf - conf.c:userns_exec_minimal:4194 - Permission denied - Running function in new user namespace failed
brauner commented 4 years ago

Can you show me findmnt, please?

4ePTuk commented 4 years ago 
findmnt
TARGET                                            SOURCE                                            FSTYPE      OPTIONS
/                                                 /dev/sda2                                         ext4        rw,relatime,errors=remount-ro,data=ordered
├─/sys                                            sysfs                                             sysfs       rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security                          securityfs                                        securityfs  rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                                tmpfs                                             tmpfs       ro,nosuid,nodev,noexec,mode=755
│ │ ├─/sys/fs/cgroup/unified                      cgroup2                                           cgroup2     rw,nosuid,nodev,noexec,relatime
│ │ ├─/sys/fs/cgroup/systemd                      cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,xattr,name=systemd
│ │ ├─/sys/fs/cgroup/pids                         cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,pids
│ │ ├─/sys/fs/cgroup/memory                       cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,memory
│ │ ├─/sys/fs/cgroup/blkio                        cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,blkio
│ │ ├─/sys/fs/cgroup/freezer                      cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,freezer
│ │ ├─/sys/fs/cgroup/cpu,cpuacct                  cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
│ │ ├─/sys/fs/cgroup/perf_event                   cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,perf_event
│ │ ├─/sys/fs/cgroup/net_cls,net_prio             cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
│ │ ├─/sys/fs/cgroup/cpuset                       cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,cpuset,clone_children
│ │ └─/sys/fs/cgroup/devices                      cgroup                                            cgroup      rw,nosuid,nodev,noexec,relatime,devices
│ ├─/sys/fs/pstore                                pstore                                            pstore      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars                     efivarfs                                          efivarfs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                                   bpf                                               bpf         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/debug                             debugfs                                           debugfs     rw,relatime
│ └─/sys/fs/fuse/connections                      fusectl                                           fusectl     rw,relatime
├─/proc                                           proc                                              proc        rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc                      systemd-1                                         autofs      rw,relatime,fd=40,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10954
│   └─/proc/sys/fs/binfmt_misc                    binfmt_misc                                       binfmt_misc rw,relatime
├─/dev                                            udev                                              devtmpfs    rw,nosuid,relatime,size=6059904k,nr_inodes=1514976,mode=755
│ ├─/dev/pts                                      devpts                                            devpts      rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/shm                                      tmpfs                                             tmpfs       rw,nosuid,nodev
│ ├─/dev/hugepages                                hugetlbfs                                         hugetlbfs   rw,relatime
│ └─/dev/mqueue                                   mqueue                                            mqueue      rw,relatime
├─/run                                            tmpfs                                             tmpfs       rw,nosuid,noexec,relatime,size=1215380k,mode=755
│ ├─/run/lock                                     tmpfs                                             tmpfs       rw,nosuid,nodev,noexec,relatime,size=5120k
│ ├─/run/rpc_pipefs                               sunrpc                                            rpc_pipefs  rw,relatime
│ └─/run/snapd/ns                                 tmpfs[/snapd/ns]                                  tmpfs       rw,nosuid,noexec,relatime,size=1215380k,mode=755
│   └─/run/snapd/ns/lxd.mnt                       nsfs[mnt:[4026532304]]                            nsfs        rw
├─/snap/core/8689                                 /dev/loop0                                        squashfs    ro,nodev,relatime
├─/snap/core/8592                                 /dev/loop1                                        squashfs    ro,nodev,relatime
├─/snap/lxd/14095                                 /dev/loop2                                        squashfs    ro,nodev,relatime
├─/snap/lxd/14066                                 /dev/loop3                                        squashfs    ro,nodev,relatime
├─/boot/efi                                       /dev/sda1                                         vfat        rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro
├─/tmp                                            tmpfs                                             tmpfs       rw,relatime
├─/srv/dev-disk-by-label-UnsafePool               /dev/sdd1                                         ext4        rw,noexec,relatime,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group
├─/srv/dev-disk-by-label-SafePool                 /dev/md127                                        ext4        rw,noexec,relatime,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group
│ └─/srv/dev-disk-by-label-SafePool/LxcContainers /dev/sda2[/var/snap/lxd/common/lxd/storage-pools] ext4        rw,relatime,errors=remount-ro,data=ordered
├─/sharedfolders/GuestShare                       /dev/sdd1[/GuestShare]                            ext4        rw,noexec,relatime,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group
├─/sharedfolders/Media                            /dev/sdd1[/Media]                                 ext4        rw,noexec,relatime,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group
├─/sharedfolders/NextCloudData                    /dev/md127[/NextCloudData]                        ext4        rw,noexec,relatime,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group
├─/sharedfolders/HomeShare                        /dev/md127[/HomeShare]                            ext4        rw,noexec,relatime,data=ordered,jqfmt=vfsv0,usrjquota=aquota.user,grpjquota=aquota.group
├─/sharedfolders/LxcContainers                    /dev/sda2[/var/snap/lxd/common/lxd/storage-pools] ext4        rw,relatime,errors=remount-ro,data=ordered
└─/var/snap/lxd/common/ns                         tmpfs                                             tmpfs       rw,relatime,size=1024k,mode=700
  ├─/var/snap/lxd/common/ns/mntns                 nsfs[mnt:[4026532304]]                            nsfs        rw
  └─/var/snap/lxd/common/ns/shmounts              nsfs[mnt:[4026532305]]                            nsfs        rw
brauner commented 4 years ago

Can you show:

ls -al /sys/fs/cgroup/unified

from inside the container, please?

4ePTuk commented 4 years ago

damn...i think that it will be problem. The only way to get inside is exec... all conrainers working without update for months. Blog is new and empty one

i thought, that exec is normal way to get inside. Because i have 24/7 access to my home server. And don't install ssh daemon on them...

brauner commented 4 years ago

No, you can either get into the container via lxc console or you can show me:

ls -al /sys/fs/cgroup/unified/

on the host and then I'll tell you which folder I need to look at :)

4ePTuk commented 4 years ago 
ls -al /sys/fs/cgroup/unified/
total 0
dr-xr-xr-x 15 root root      0 Mar 30 14:19 .
drwxr-xr-x 13 root root    340 Mar 30 14:03 ..
-r--r--r--  1 root root      0 Mar 30 14:19 cgroup.controllers
-rw-r--r--  1 root root      0 Mar 30 14:03 cgroup.procs
-rw-r--r--  1 root root      0 Mar 30 14:19 cgroup.subtree_control
drwxr-xr-x  2 root root      0 Mar 30 14:19 init.scope
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.Blog
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.Emby
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.Gitea
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.NextCloud
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.NginxReverseProxy
drwxrwxr-x  3 root 1000000   0 Mar 30 14:03 lxc.payload.Blog
drwxrwxr-x  2 root 1000000   0 Mar 30 14:03 lxc.payload.Emby
drwxrwxr-x  2 root 1000000   0 Mar 30 14:03 lxc.payload.Gitea
drwxrwxr-x  2 root 1000000   0 Mar 30 14:03 lxc.payload.NextCloud
drwxrwxr-x  2 root 1000000   0 Mar 30 14:03 lxc.payload.NginxReverseProxy
drwxr-xr-x 57 root root      0 Mar 30 14:19 system.slice
drwxr-xr-x  2 root root      0 Mar 30 14:03 user.slice

How to connect to Blog using console? lxc-console -n Blog ? It says that Blog is not running

brauner commented 4 years ago

Ok, can you show:

ls -al /sys/fs/cgroup/unified/lxc.payload.Blog

please, that should be the container you showed the log from, right?

4ePTuk commented 4 years ago 
root@openmediavault:~# ls -al /sys/fs/cgroup/unified/lxc.payload.Blog
total 0
drwxrwxr-x  3 root    1000000 0 Mar 30 14:03 .
dr-xr-xr-x 15 root    root    0 Mar 30 14:19 ..
drwxr-xr-x  2 1000000 1000000 0 Mar 30 14:03 .lxc
-r--r--r--  1 root    root    0 Mar 30 14:45 cgroup.controllers
-r--r--r--  1 root    root    0 Mar 30 14:45 cgroup.events
-rw-rw-r--  1 root    1000000 0 Mar 30 14:03 cgroup.procs
-rw-rw-r--  1 root    1000000 0 Mar 30 14:03 cgroup.subtree_control

yes, this is brand new container. But other have same error

brauner commented 4 years ago

Huh, can you show:

cat /sys/fs/cgroup/unified/lxc.payload.Blog/cgroup.procs
ls -al /sys/fs/cgroup/unified/lxc.payload.Blog/.lxc

please?

4ePTuk commented 4 years ago 
root@openmediavault:~# cat /sys/fs/cgroup/unified/lxc.payload.Blog/cgroup.procs
1536
1396
1498
1535
1795
1823
root@openmediavault:~# ls -al /sys/fs/cgroup/unified/lxc.payload.Blog/.lxc
total 0
drwxr-xr-x 2 1000000 1000000 0 Mar 30 14:03 .
drwxrwxr-x 3 root    1000000 0 Mar 30 14:03 ..
-r--r--r-- 1 1000000 1000000 0 Mar 30 14:03 cgroup.controllers
-r--r--r-- 1 1000000 1000000 0 Mar 30 14:03 cgroup.events
-rw-r--r-- 1 1000000 1000000 0 Mar 30 14:03 cgroup.procs
-rw-r--r-- 1 1000000 1000000 0 Mar 30 14:03 cgroup.subtree_control

=( Do you have any idea?

brauner commented 4 years ago 
ls -al /sys/fs/cgroup/unified/
total 0
dr-xr-xr-x 15 root root      0 Mar 30 14:19 .
drwxr-xr-x 13 root root    340 Mar 30 14:03 ..
-r--r--r--  1 root root      0 Mar 30 14:19 cgroup.controllers
-rw-r--r--  1 root root      0 Mar 30 14:03 cgroup.procs
-rw-r--r--  1 root root      0 Mar 30 14:19 cgroup.subtree_control
drwxr-xr-x  2 root root      0 Mar 30 14:19 init.scope
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.Blog
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.Emby
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.Gitea
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.NextCloud
drwxr-xr-x  2 root root      0 Mar 30 14:19 lxc.monitor.NginxReverseProxy
drwxrwxr-x  3 root 1000000   0 Mar 30 14:03 lxc.payload.Blog
drwxrwxr-x  2 root 1000000   0 Mar 30 14:03 lxc.payload.Emby
drwxrwxr-x  2 root 1000000   0 Mar 30 14:03 lxc.payload.Gitea
drwxrwxr-x  2 root 1000000   0 Mar 30 14:03 lxc.payload.NextCloud
drwxrwxr-x  2 root 1000000   0 Mar 30 14:03 lxc.payload.NginxReverseProxy
drwxr-xr-x 57 root root      0 Mar 30 14:19 system.slice
drwxr-xr-x  2 root root      0 Mar 30 14:03 user.slice

How to connect to Blog using console? lxc-console -n Blog ? It says that Blog is not running

lxc console Blog
4ePTuk commented 4 years ago

but without exec i can't set default passwd right?

brauner commented 4 years ago

What container is that you're running? I've just created a new Debian stretch vm, install lxd from the snap and ran an ubuntu container and attached to it just fine

brauner commented 4 years ago

And what systemd version are you running on the host:

systemctl --version

?

4ePTuk commented 4 years ago 
root@openmediavault:~# systemctl --version
systemd 241 (241)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

root@openmediavault:~# lxc list
+-------------------+---------+-----------------------+------+-----------+-----------+
|       NAME        |  STATE  |         IPV4          | IPV6 |   TYPE    | SNAPSHOTS |
+-------------------+---------+-----------------------+------+-----------+-----------+
| Blog              | RUNNING | 192.168.31.120 (eth0) |      | CONTAINER | 0         |
+-------------------+---------+-----------------------+------+-----------+-----------+
| Cups              | STOPPED |                       |      | CONTAINER | 0         |
+-------------------+---------+-----------------------+------+-----------+-----------+
| Emby              | RUNNING | 192.168.31.53 (eth0)  |      | CONTAINER | 0         |
+-------------------+---------+-----------------------+------+-----------+-----------+
| Gitea             | RUNNING | 192.168.31.52 (eth0)  |      | CONTAINER | 6         |
+-------------------+---------+-----------------------+------+-----------+-----------+
| NextCloud         | RUNNING | 192.168.31.57 (eth0)  |      | CONTAINER | 0         |
+-------------------+---------+-----------------------+------+-----------+-----------+
| NginxReverseProxy | RUNNING | 192.168.31.51 (eth0)  |      | CONTAINER | 0         |
+-------------------+---------+-----------------------+------+-----------+-----------+
| test              | STOPPED |                       |      | CONTAINER | 0         |
+-------------------+---------+-----------------------+------+-----------+-----------+

All containters have same error

brauner commented 4 years ago

Why do you have systemd 241 installed?

brauner commented 4 years ago

Or rather, how?

4ePTuk commented 4 years ago

i don't know...never touched it should i update it to latest?

brauner commented 4 years ago

i don't know...never touched it should i update it to latest?

No, let me try and reproduce with this setup.

4ePTuk commented 4 years ago

thanks...i'm close to reinstall system to get it work (

brauner commented 4 years ago

Ok, managed to reproduce this and it's sucky but I have an idea.

4ePTuk commented 4 years ago

I'm full of attention =)

brauner commented 4 years ago

The issue is with the kernel you're using. On this kernel the restrictions to move processes between cgroups are different than they are on newer kernels. Specifically, you're running into the following check:

    if (!uid_eq(cred->euid, GLOBAL_ROOT_UID) &&
        !uid_eq(cred->euid, tcred->uid) &&
        !uid_eq(cred->euid, tcred->suid))
        ret = -EACCES;

which dictates that in order to move a process into a cgroup you either need to be global root (no restrictions apply) or the effective uid of the process trying to move the process and the {saved}uid of the process that is supposed to be mvoed need to be identical. The new attaching logic we did doesn't fulfill this criterion for various reasons. I can likely fix this but I'm starting to think about placing a requirement on the kernel version for which we guarantee cgroup2 support. Mainly because cgroup2 has changed quite a bit.

4ePTuk commented 4 years ago

so, in this condition what can you advice? rollback on early version of lxd? i can't update disto right now. How can i login into container with lxd-console if there is no passwd?

brauner commented 4 years ago

I'll send a fix soon and then @stgraber will cherry-pick it into the snap and you should have it in a few hours (@stgraber, right?).

4ePTuk commented 4 years ago

If the Qt conference had not been rescheduled into USA and it remained in Berlin i could buy you some beer ) Thank you

stgraber commented 4 years ago

Yep, cherry-pick is pretty quick

4ePTuk commented 4 years ago

what should i do after this? snap refresh lxc to candidate?