search

nestybox / sysbox

An open-source, next-generation "runc" that empowers rootless containers to run workloads such as Systemd, Docker, Kubernetes, just like VMs.
Apache License 2.0
2.7k stars 151 forks source link

ID mapping problem inside container #812

Open ffabreti opened 2 months ago

ffabreti commented 2 months ago

Thanks for sysbox, it's great!

I'm having an issue with files inside sysbox container appearing with nobody:nogroup. Strangely, not all files, but some. I have read several issues here, and i'm looking for culprits.

I have upgraded my host Ubuntu 22.04.3 from kernel 5.15.0-113 to 6.5.0-41 trying to solve the problem to no avail.

I'm looking at LVM now, because my host /var/lib/docker is mounted on a LVM volume (not sure if this still applies, I've seen some old issues)

here are some info I think you should ask, some sensitive or noise info are omitted:

HOST INFO:


# lsb_release -a
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

# uname -a
Linux 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

lsmod | grep shiftfs
<not loaded>

# lsblk -f
NAME                      FSTYPE      FSVER    LABEL             MOUNTPOINTS
sda                                                              
└─sda1                    LVM2_member LVM2 001                   
  └─vld-lvvld             ext4        1.0      lvm-varlibdocker  /var/lib/docker
sdc                                                              
├─sdc1                                                           
├─sdc2                    ext4        1.0                        /boot
└─sdc3                    LVM2_member LVM2 001                   
  └─ubuntu--vg-ubuntu--lv ext4        1.0                        /

# cat /etc/default/grub
GRUB_CMDLINE_LINUX="ipv6.disable=1"

# systemctl status sysbox-mgr
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Starting ..."
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Sysbox data root: /var/lib/sysbox"
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Shiftfs module found in kernel: no"
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Shiftfs works properly: no"
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Shiftfs-on-overlayfs works properly: no"
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="ID-mapped mounts supported by kernel: yes"
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Overlayfs on ID-mapped mounts supported by kernel: yes"
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Operating in system container mode."
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Inner container image preloading enabled."
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Listening on /run/sysbox/sysmgr.sock"
Jul 01 20:20:58 sysbox-mgr[31471]: level=info msg="Ready ..."

# systemctl status sysbox-fs -n 20
Jul 01 20:58:03 sysbox-fs[31492]: time="2024-07-01 20:58:03" level=warning msg="Received seccompNotifMsg generated by unknown container: 6ac160a4b546"
Jul 01 20:58:03 sysbox-fs[31492]: time="2024-07-01 20:58:03" level=warning msg="Received seccompNotifMsg generated by unknown container: 6ac160a4b546"
Jul 01 20:58:03 sysbox-fs[31492]: time="2024-07-01 20:58:03" level=warning msg="Received seccompNotifMsg generated by unknown container: 6ac160a4b546"
Jul 01 20:58:03 sysbox-fs[31492]: time="2024-07-01 20:58:03" level=warning msg="Received seccompNotifMsg generated by unknown container: 6ac160a4b546"
Jul 01 20:58:03 sysbox-fs[31492]: time="2024-07-01 20:58:03" level=warning msg="Received seccompNotifMsg generated by unknown container: 6ac160a4b546"
Jul 01 20:58:03 sysbox-fs[31492]: time="2024-07-01 20:58:03" level=warning msg="Received seccompNotifMsg generated by unknown container: 6ac160a4b546"
Jul 02 08:24:42 sysbox-fs[31492]: time="2024-07-02 08:24:42" level=info msg="Container pre-registration completed: id = 6ac160a4b546"
Jul 02 08:24:42 sysbox-fs[31492]: time="2024-07-02 08:24:42" level=info msg="Container registration completed: id = 6ac160a4b546, initPid = 64477, uid:gid = 165536:165536"
Jul 02 08:33:58 sysbox-fs[31492]: time="2024-07-02 08:33:58" level=info msg="Container pre-registration completed: id = 0757f1a11a71"
Jul 02 08:33:58 sysbox-fs[31492]: time="2024-07-02 08:33:58" level=info msg="Container registration completed: id = 0757f1a11a71, initPid = 70628, uid:gid = 165536:165536"

# systemctl status sysbox -n 20
Jul 01 20:20:58 sh[31502]: sysbox-runc
Jul 01 20:20:58 sh[31502]:         edition:         Community Edition (CE)
Jul 01 20:20:58 sh[31502]:         version:         0.6.2
Jul 01 20:20:58 sh[31502]:         oci-specs:         1.0.2-dev
Jul 01 20:20:58 sh[31508]: sysbox-mgr
Jul 01 20:20:58 sh[31508]:         edition:         Community Edition (CE)
Jul 01 20:20:58 sh[31508]:         version:         0.6.2
Jul 01 20:20:58 sh[31513]: sysbox-fs
Jul 01 20:20:58 sh[31513]:         edition:         Community Edition (CE)
Jul 01 20:20:58 sh[31513]:         version:         0.6.2

# cat /etc/docker/daemon.json
{
    "bip": "192.168.60.1/27",
    "default-address-pools": [
        {
            "base": "192.168.61.0/24",
            "size": 27
        }
    ],
    "ip-masq": true,
    "ipv6": false,
    "default-runtime": "sysbox-runc",
    "runtimes": {
        "sysbox-runc": {
            "path": "/usr/bin/sysbox-runc"
        }
    }
}

# docker info
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
Server:
 Storage Driver: overlay2

# alias dps='docker ps -a       --format '\''table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}'\'''
# dps
CONTAINER ID   NAMES        STATUS        PORTS
6ac160a4b546   hmapp03      Up 4 hours    5000-5050->5000-5050/tcp, 5051-5100/tcp, 2211->22/tcp

#docker run for container:
docker run \
--runtime=sysbox-runc \
--hostname hmapp03 \
--name hmapp03 \
--restart=unless-stopped \
--mount source=varlibdocker-hmapp03-v1,target=/var/lib/docker \
--ip 192.168.61.5 \
--network br-hmnet \
-p 2211:22 \
-p 5000-5050:5000-5050 \
--detach \
ubuntu-jammy-systemd-docker:v4 <this is a custom image>

INSIDE SYSBOX CONTAINER INFO (HMAPP03):

I've noticed on hmapp03, var/lib/docker is not idmapped!
#mount | grep docker 
/dev/mapper/vld-lvvld on /var/lib/docker type ext4 (rw,relatime)

# findmnt -J 
# docker exec -it hmapp03  findmnt -J
{
   "filesystems": [
      {
         "target": "/",
         "source": "overlay",
         "fstype": "overlay",
         "options": "rw,relatime,lowerdir=/var/lib/docker/overlay2/l/YKFESOSMHM6Z6CQID7P4P4H3DN:/var/lib/docker/overlay2/l/DAEVDBOZUPJINUQRLNVL2AXQRZ:/var/lib/docker/overlay2/l/2Y25TM5F7W3MDXAFE4RWSXYBYA:/var/lib/docker/overlay2/l/VQJ6BA3564D65XP2YOLXMM2XPO:/var/lib/docker/overlay2/l/AMSLSHZXUEZRUVE6S76C7ITFXK:/var/lib/docker/overlay2/l/MTD6J762Q4K6XBMIDH65CT55Z3:/var/lib/docker/overlay2/l/32OYFFIY5KGNXMKSYJAG636LDT:/var/lib/docker/overlay2/l/D3BINRCBRXO2PNY3WVUIWPYX2U:/var/lib/docker/overlay2/l/5FNLNOK5VPY3C7HI5HGCMO4B2J:/var/lib/docker/overlay2/l/PJX37BIXPOV7QULXJFHIWIEB2E:/var/lib/docker/overlay2/l/FOCPSCVXCLWATQBHZCPOD6H2RN:/var/lib/docker/overlay2/l/5IWJDNKDG64HJFOPLRNRYERE4E,upperdir=/var/lib/docker/overlay2/87d1553acd6027d1b0a47d459dd85f5e52ed71cfa6dea415640acee49872bcbe/diff,workdir=/var/lib/docker/overlay2/87d1553acd6027d1b0a47d459dd85f5e52ed71cfa6dea415640acee49872bcbe/work,nouserxattr",
         "children": [
            {
               "target": "/sys",
               "source": "sysfs",
               "fstype": "sysfs",
               "options": "rw,nosuid,nodev,noexec,relatime",
               "children": [
                  {
                     "target": "/sys/firmware",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "ro,relatime,uid=165536,gid=165536,inode64"
                  },{
                     "target": "/sys/fs/cgroup",
                     "source": "cgroup",
                     "fstype": "cgroup2",
                     "options": "rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot"
                  },{
                     "target": "/sys/devices/virtual",
                     "source": "sysboxfs[/sys/devices/virtual]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  },{
                     "target": "/sys/kernel",
                     "source": "sysboxfs[/sys/kernel]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  },{
                     "target": "/sys/module/nf_conntrack/parameters",
                     "source": "sysboxfs[/sys/module/nf_conntrack/parameters]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  }
               ]
            },{
               "target": "/proc",
               "source": "proc",
               "fstype": "proc",
               "options": "rw,nosuid,nodev,noexec,relatime",
               "children": [
                  {
                     "target": "/proc/bus",
                     "source": "proc[/bus]",
                     "fstype": "proc",
                     "options": "ro,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/proc/fs",
                     "source": "proc[/fs]",
                     "fstype": "proc",
                     "options": "ro,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/proc/irq",
                     "source": "proc[/irq]",
                     "fstype": "proc",
                     "options": "ro,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/proc/sysrq-trigger",
                     "source": "proc[/sysrq-trigger]",
                     "fstype": "proc",
                     "options": "ro,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/proc/acpi",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "ro,relatime,uid=165536,gid=165536,inode64"
                  },{
                     "target": "/proc/keys",
                     "source": "udev[/null]",
                     "fstype": "devtmpfs",
                     "options": "rw,nosuid,relatime,size=3996644k,nr_inodes=999161,mode=755,inode64"
                  },{
                     "target": "/proc/timer_list",
                     "source": "udev[/null]",
                     "fstype": "devtmpfs",
                     "options": "rw,nosuid,relatime,size=3996644k,nr_inodes=999161,mode=755,inode64"
                  },{
                     "target": "/proc/scsi",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "ro,relatime,uid=165536,gid=165536,inode64"
                  },{
                     "target": "/proc/swaps",
                     "source": "sysboxfs[/proc/swaps]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  },{
                     "target": "/proc/sys",
                     "source": "sysboxfs[/proc/sys]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  },{
                     "target": "/proc/uptime",
                     "source": "sysboxfs[/proc/uptime]",
                     "fstype": "fuse",
                     "options": "rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other"
                  }
               ]
            },{
               "target": "/dev",
               "source": "tmpfs",
               "fstype": "tmpfs",
               "options": "rw,nosuid,size=65536k,mode=755,uid=165536,gid=165536,inode64",
               "children": [
                  {
                     "target": "/dev/mqueue",
                     "source": "mqueue",
                     "fstype": "mqueue",
                     "options": "rw,nosuid,nodev,noexec,relatime"
                  },{
                     "target": "/dev/pts",
                     "source": "devpts",
                     "fstype": "devpts",
                     "options": "rw,nosuid,noexec,relatime,gid=165541,mode=620,ptmxmode=666"
                  },{
                     "target": "/dev/shm",
                     "source": "shm",
                     "fstype": "tmpfs",
                     "options": "rw,nosuid,nodev,noexec,relatime,size=65536k,uid=165536,gid=165536,inode64"
                  },{
                     "target": "/dev/null",
                     "source": "udev[/null]",
                     "fstype": "devtmpfs",
                     "options": "rw,nosuid,relatime,size=3996644k,nr_inodes=999161,mode=755,inode64"
                  },{
                     "target": "/dev/random",
                     "source": "udev[/random]",
                     "fstype": "devtmpfs",
                     "options": "rw,nosuid,relatime,size=3996644k,nr_inodes=999161,mode=755,inode64"
                  },{
                     "target": "/dev/kmsg",
                     "source": "udev[/null]",
                     "fstype": "devtmpfs",
                     "options": "rw,nosuid,relatime,size=3996644k,nr_inodes=999161,mode=755,inode64"
                  },{
                     "target": "/dev/full",
                     "source": "udev[/full]",
                     "fstype": "devtmpfs",
                     "options": "rw,nosuid,relatime,size=3996644k,nr_inodes=999161,mode=755,inode64"
                  },{
                     "target": "/dev/tty",
                     "source": "udev[/tty]",
                     "fstype": "devtmpfs",
                     "options": "rw,nosuid,relatime,size=3996644k,nr_inodes=999161,mode=755,inode64"
                  },{
                     "target": "/dev/zero",
                     "source": "udev[/zero]",
                     "fstype": "devtmpfs",
                     "options": "rw,nosuid,relatime,size=3996644k,nr_inodes=999161,mode=755,inode64"
                  },{
                     "target": "/dev/urandom",
                     "source": "udev[/urandom]",
                     "fstype": "devtmpfs",
                     "options": "rw,nosuid,relatime,size=3996644k,nr_inodes=999161,mode=755,inode64"
                  }
               ]
            },{
               "target": "/run",
               "source": "tmpfs",
               "fstype": "tmpfs",
               "options": "rw,nosuid,nodev,relatime,size=65536k,mode=755,uid=165536,gid=165536,inode64",
               "children": [
                  {
                     "target": "/run/lock",
                     "source": "tmpfs",
                     "fstype": "tmpfs",
                     "options": "rw,nosuid,nodev,noexec,relatime,size=4096k,uid=165536,gid=165536,inode64"
                  },{
                     "target": "/run/docker/netns/5aee14bec16d",
                     "source": "nsfs[net:[4026532823]]",
                     "fstype": "nsfs",
                     "options": "rw"
                  },{
                     "target": "/run/docker/netns/9b8c45925f2a",
                     "source": "nsfs[net:[4026532865]]",
                     "fstype": "nsfs",
                     "options": "rw"
                  },{
                     "target": "/run/docker/netns/0379afe1348b",
                     "source": "nsfs[net:[4026532982]]",
                     "fstype": "nsfs",
                     "options": "rw"
                  },{
                     "target": "/run/docker/netns/3d1e73bef29a",
                     "source": "nsfs[net:[4026533236]]",
                     "fstype": "nsfs",
                     "options": "rw"
                  },{
                     "target": "/run/docker/netns/7996ac6b4713",
                     "source": "nsfs[net:[4026532766]]",
                     "fstype": "nsfs",
                     "options": "rw"
                  }
               ]
            },{
               "target": "/var/lib/docker",
               "source": "/dev/mapper/vld-lvvld[/volumes/varlibdocker-hmapp03-v1/_data]",
               "fstype": "ext4",
               "options": "rw,relatime",
               "children": [
                  {
                     "target": "/var/lib/docker/overlay2/794da6b1730f55ec9f529b21bcac141bc5e084873c921cbfd216b65e9c79ef5a/merged",
                     "source": "overlay",
                     "fstype": "overlay",
                     "options": "rw,relatime,lowerdir=/var/lib/docker/overlay2/l/IJGECUQDZZT4B7SAZPN3W5XF57:/var/lib/docker/overlay2/l/242FAZFGRCG5GEL4PI5PSBPPN5:/var/lib/docker/overlay2/l/6E3UMHBHAICZUSIB2K6GPEASGF:/var/lib/docker/overlay2/l/KVIZ37QFA3U32ZOX7NB6664UKF:/var/lib/docker/overlay2/l/SKQZE26LMQUYPOQBRBHZRMVTJY:/var/lib/docker/overlay2/l/3ORJU3I563L3HVMHAODIYSXWV7:/var/lib/docker/overlay2/l/NO5VHYUJX7B4DCOW4MK6XSBZFL:/var/lib/docker/overlay2/l/MDYRSYD6VNSGKIJTKO6OVF3SOL:/var/lib/docker/overlay2/l/O3QRDM6V7YN7TEG37FRFCOIP5D:/var/lib/docker/overlay2/l/DX52OBUHLV5N4QXFLVNPJO5SMV:/var/lib/docker/overlay2/l/3ZZMO7XU6UJESPUIYZI2WDU5W3:/var/lib/docker/overlay2/l/KTAVS67FJALBGNFQTSN4HFKVRJ:/var/lib/docker/overlay2/l/SCDBG2QVH75IQET75ADJJUQ7Y5:/var/lib/docker/overlay2/l/V4VEZI5ANZOEPDR57VM52JEJWA:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/794da6b1730f55ec9f529b21bcac141bc5e084873c921cbfd216b65e9c79ef5a/diff,workdir=/var/lib/docker/overlay2/794da6b1730f55ec9f529b21bcac141bc5e084873c921cbfd216b65e9c79ef5a/work,redirect_dir=nofollow,userxattr"
                  },{
                     "target": "/var/lib/docker/overlay2/2d7613e2f6be28ab4ed42850a587f8c0d70ecc0dd51444e16265bf56ca361739/merged",
                     "source": "overlay",
                     "fstype": "overlay",
                     "options": "rw,relatime,lowerdir=/var/lib/docker/overlay2/l/2IGYB2PAURIM46PDXEMMSOQDTE:/var/lib/docker/overlay2/l/QCEQNQWAJBSFXITCBNPYXHDW4O:/var/lib/docker/overlay2/l/32WN5ZDZWIVELHFYZPP2QJM7Q3:/var/lib/docker/overlay2/l/N2ST2NSFJ7L5SU7ZBJR4HODJIY:/var/lib/docker/overlay2/l/FVARIARVXMQRBJBTSFL56FQ53H:/var/lib/docker/overlay2/l/NFBR7YWBXIPDOGCLSXGOXTJGFM:/var/lib/docker/overlay2/l/H3LH7DM7B32POFRQRVNKWZFFJB:/var/lib/docker/overlay2/l/XJF5XW6JMEHKIJI7NTFFCJYPUO:/var/lib/docker/overlay2/l/VO76EUOKXH5NOQDFHQJPFXXIXN:/var/lib/docker/overlay2/l/GE4P3FMSVD2SSAILR2DCLRORNL:/var/lib/docker/overlay2/l/3ZZMO7XU6UJESPUIYZI2WDU5W3:/var/lib/docker/overlay2/l/KTAVS67FJALBGNFQTSN4HFKVRJ:/var/lib/docker/overlay2/l/SCDBG2QVH75IQET75ADJJUQ7Y5:/var/lib/docker/overlay2/l/V4VEZI5ANZOEPDR57VM52JEJWA:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/2d7613e2f6be28ab4ed42850a587f8c0d70ecc0dd51444e16265bf56ca361739/diff,workdir=/var/lib/docker/overlay2/2d7613e2f6be28ab4ed42850a587f8c0d70ecc0dd51444e16265bf56ca361739/work,redirect_dir=nofollow,userxattr"
                  },{
                     "target": "/var/lib/docker/overlay2/325a7f765a5afbfeaf78fc926eb9bd19dba57d7368f6c500cc7c5a5492be9c17/merged",
                     "source": "overlay",
                     "fstype": "overlay",
                     "options": "rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LKCRDQGZOJVHXXYH67WDNORUEF:/var/lib/docker/overlay2/l/FJSIHV5ATS7P6OW3PF6K6OBVXR:/var/lib/docker/overlay2/l/GCX24ICVZTEC4MHXCOC5RG3JVR:/var/lib/docker/overlay2/l/T5JXXYAXSJYAG324V2HPPWWXGZ:/var/lib/docker/overlay2/l/4H27II67CLWEENJRMQA2C2YANQ:/var/lib/docker/overlay2/l/HWV5TOLZY7EJALI4SZPZSSXCBZ:/var/lib/docker/overlay2/l/JXZBLYGCEZ2VIWJWWIQWABPJ63:/var/lib/docker/overlay2/l/42ZBEEUDNWCHKNB2YCEABZN5BT:/var/lib/docker/overlay2/l/C2ZACQXFHC3Z75FQKEFRLP5TIA:/var/lib/docker/overlay2/l/5HSBWXDIS37PWGRJH7HSXXFVOD:/var/lib/docker/overlay2/l/MADUBJJRSJ6EACBNCEYVXYECRI:/var/lib/docker/overlay2/l/YMSFVFZWO24RX4JFBURYVH2HL6:/var/lib/docker/overlay2/l/BFBYJEBZ5WYDIJ6EB5JAIZISY6:/var/lib/docker/overlay2/l/SEH7KXYAZNLG5DMA3MO55ST2ZT:/var/lib/docker/overlay2/l/ZA7IM4U45WBYDZBXXC3WWHXP4P,upperdir=/var/lib/docker/overlay2/325a7f765a5afbfeaf78fc926eb9bd19dba57d7368f6c500cc7c5a5492be9c17/diff,workdir=/var/lib/docker/overlay2/325a7f765a5afbfeaf78fc926eb9bd19dba57d7368f6c500cc7c5a5492be9c17/work,redirect_dir=nofollow,userxattr"
                  },{
                     "target": "/var/lib/docker/overlay2/06ed7e3c8b72c21557fc1b0de430d03ef5f5dec0ddf96baa160fc50fe6dcb08b/merged",
                     "source": "overlay",
                     "fstype": "overlay",
                     "options": "rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SWQBQKDUOHWKBHKZFI53TVXFHF:/var/lib/docker/overlay2/l/JMLUD7ZMFMEC6GB7W2SL6ZGBPM:/var/lib/docker/overlay2/l/QLOX7MTGZVLC3WCVRM2O65WE6T:/var/lib/docker/overlay2/l/QXBMMDMDAXDHPXCR6Q5IAZCACC:/var/lib/docker/overlay2/l/RMEA2WRCJB5BFMUMV65I6FP7D5:/var/lib/docker/overlay2/l/GI65S43RQ7OQ34S4XT3YK6DH5T:/var/lib/docker/overlay2/l/J32RM357H3JNJU3PQATZRILPB3:/var/lib/docker/overlay2/l/F3DZTXATKSVZRDU7TOE25SAOAV:/var/lib/docker/overlay2/l/ITOLGDS3JIP7DDRCLC43W6DLOT:/var/lib/docker/overlay2/l/JMVQZOGQAZYBAAWOYRXIISX4V6:/var/lib/docker/overlay2/l/TXRBEOBOSX2UZHPASF3IPLOCVX,upperdir=/var/lib/docker/overlay2/06ed7e3c8b72c21557fc1b0de430d03ef5f5dec0ddf96baa160fc50fe6dcb08b/diff,workdir=/var/lib/docker/overlay2/06ed7e3c8b72c21557fc1b0de430d03ef5f5dec0ddf96baa160fc50fe6dcb08b/work,redirect_dir=nofollow,userxattr"
                  },{
                     "target": "/var/lib/docker/overlay2/c89d9a69088e05f433c3fa272634e0b22ae9ee3adc3d214e65ad3b7f4c165dfe/merged",
                     "source": "overlay",
                     "fstype": "overlay",
                     "options": "rw,relatime,lowerdir=/var/lib/docker/overlay2/l/KROVCC5UCY3GDH7PJZBKYKS2JH:/var/lib/docker/overlay2/l/MFYO2BVZIEZOUJWWTECHGXOVBN:/var/lib/docker/overlay2/l/AFNZU4MF5LDLQOA7QVCZAMUKCA:/var/lib/docker/overlay2/l/FVZ5IRPEYTB4H7EX23LRFJDPAJ:/var/lib/docker/overlay2/l/5O2OYMZUGMUY2VFEOH25MI6JVF:/var/lib/docker/overlay2/l/3EJQUHWAUWLLVCQNIWARMQ6ECT:/var/lib/docker/overlay2/l/TATTM3IMPFQ7V4HEZRI53B3U3W:/var/lib/docker/overlay2/l/XQOD42Y55SS2PWY5QZ357H67LL:/var/lib/docker/overlay2/l/IR3QP77BLA7BIIAUBQZCEFVVE7:/var/lib/docker/overlay2/l/MADUBJJRSJ6EACBNCEYVXYECRI:/var/lib/docker/overlay2/l/YMSFVFZWO24RX4JFBURYVH2HL6:/var/lib/docker/overlay2/l/BFBYJEBZ5WYDIJ6EB5JAIZISY6:/var/lib/docker/overlay2/l/SEH7KXYAZNLG5DMA3MO55ST2ZT:/var/lib/docker/overlay2/l/ZA7IM4U45WBYDZBXXC3WWHXP4P,upperdir=/var/lib/docker/overlay2/c89d9a69088e05f433c3fa272634e0b22ae9ee3adc3d214e65ad3b7f4c165dfe/diff,workdir=/var/lib/docker/overlay2/c89d9a69088e05f433c3fa272634e0b22ae9ee3adc3d214e65ad3b7f4c165dfe/work,redirect_dir=nofollow,userxattr"
                  }
               ]
            },{
               "target": "/etc/resolv.conf",
               "source": "/dev/mapper/vld-lvvld[/containers/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa/resolv.conf]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped"
            },{
               "target": "/etc/hostname",
               "source": "/dev/mapper/vld-lvvld[/containers/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa/hostname]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped"
            },{
               "target": "/etc/hosts",
               "source": "/dev/mapper/vld-lvvld[/containers/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa/hosts]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped"
            },{
               "target": "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs",
               "source": "/dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/containerd/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped"
            },{
               "target": "/var/lib/rancher/k3s",
               "source": "/dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/rancher-k3s/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped"
            },{
               "target": "/var/lib/rancher/rke2",
               "source": "/dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/rancher-rke2/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped"
            },{
               "target": "/var/lib/kubelet",
               "source": "/dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/kubelet/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped"
            },{
               "target": "/var/lib/k0s",
               "source": "/dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/k0s/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped"
            },{
               "target": "/var/lib/buildkit",
               "source": "/dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/buildkit/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]",
               "fstype": "ext4",
               "options": "rw,relatime,idmapped"
            },{
               "target": "/usr/src",
               "source": "/dev/mapper/ubuntu--vg-ubuntu--lv[/usr/src]",
               "fstype": "ext4",
               "options": "ro,relatime,idmapped",
               "children": [
                  {
                     "target": "/usr/src/linux-headers-6.5.0-41-generic",
                     "source": "/dev/mapper/ubuntu--vg-ubuntu--lv[/usr/src/linux-headers-6.5.0-41-generic]",
                     "fstype": "ext4",
                     "options": "ro,relatime,idmapped"
                  }
               ]
            },{
               "target": "/usr/lib/modules/6.5.0-41-generic",
               "source": "/dev/mapper/ubuntu--vg-ubuntu--lv[/usr/lib/modules/6.5.0-41-generic]",
               "fstype": "ext4",
               "options": "ro,relatime,idmapped"
            }
         ]
      }
   ]
}

Here is the nobody:nogroup problem

# docker exec -it hmapp03  ls -la /etc
total 852
drwxr-xr-x 1 root   root     4096 Jun 12 23:44 .
drwxr-xr-x 1 root   root     4096 Jun 25 21:00 ..
-rw------- 1 root   root        0 Oct  4  2023 .pwd.lock
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:32 X11
-rw-r--r-- 1 root   root     3028 Oct  4  2023 adduser.conf
drwxr-xr-x 1 root   root     4096 Jun 13 00:13 alternatives
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:33 apparmor
drwxr-xr-x 1 root   root     4096 May 28 18:47 apparmor.d
drwxr-xr-x 1 root   root     4096 Oct  4  2023 apt
-rw-r--r-- 1 root   root     2319 Jan  6  2022 bash.bashrc
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 18:21 bash_completion.d
-rw-r--r-- 1 root   root      367 Dec 16  2020 bindresvport.blacklist
drwxr-xr-x 1 nobody nogroup  4096 Sep 19  2023 binfmt.d
drwxr-xr-x 1 nobody nogroup  4096 Nov 30  2023 ca-certificates
-rw-r--r-- 1 root   root     5892 Nov 30  2023 ca-certificates.conf
drwxr-xr-x 1 nobody nogroup  4096 Oct  4  2023 cloud
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:33 containerd
drwxr-xr-x 1 root   root     4096 Jan  7 17:33 cron.d
drwxr-xr-x 1 root   root     4096 Jan  7 17:33 cron.daily
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:33 cron.hourly
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:33 cron.monthly
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:33 cron.weekly
-rw-r--r-- 1 root   root     1136 Mar 23  2022 crontab
drwxr-xr-x 1 nobody nogroup  4096 Nov 30  2023 dbus-1
-rw-r--r-- 1 root   root     2969 Feb 20  2022 debconf.conf
-rw-r--r-- 1 root   root       13 Aug 22  2021 debian_version
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 18:21 default
-rw-r--r-- 1 root   root      604 Sep 15  2018 deluser.conf
drwxr-xr-x 1 nobody nogroup  4096 Nov 30  2023 depmod.d
drwxr-xr-x 1 root   root     4096 Jan  7 18:21 docker
drwxr-xr-x 1 nobody nogroup  4096 Oct  4  2023 dpkg
-rw-r--r-- 1 root   root      685 Jan  8  2022 e2scrub.conf
-rw-r--r-- 1 root   root      106 Oct  4  2023 environment
-rw-r--r-- 1 root   root     1816 Dec 27  2019 ethertypes
-rw-r--r-- 1 root   root       37 Oct  4  2023 fstab
-rw-r--r-- 1 root   root     2584 Feb  3  2022 gai.conf
-rw-r--r-- 1 root   root      903 Jun 11 13:33 group
-rw-r--r-- 1 root   root      889 Jun 11 13:33 group-
-rw-r----- 1 root   shadow    756 Jun 11 13:33 gshadow
-rw-r----- 1 root   shadow    742 Jun 11 13:33 gshadow-
drwxr-xr-x 1 nobody nogroup  4096 Feb 21  2022 gss
-rw-r--r-- 1 root   root       92 Oct 15  2021 host.conf
-rw-r--r-- 1 root   root        8 Jul  2 11:24 hostname
-rw-r--r-- 1 root   root      171 Jul  2 11:24 hosts
-rw-r--r-- 1 nobody nogroup   411 Jan  7 18:21 hosts.allow
-rw-r--r-- 1 nobody nogroup   711 Jan  7 18:21 hosts.deny
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:33 init
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 18:21 init.d
-rw-r--r-- 1 root   root     1748 Jan  6  2022 inputrc
drwxr-xr-x 1 nobody nogroup  4096 Nov 30  2023 iproute2
-rw-r--r-- 1 root   root       26 Aug  2  2023 issue
-rw-r--r-- 1 root   root       19 Aug  2  2023 issue.net
drwxr-xr-x 1 root   root     4096 Nov 30  2023 kernel
-rw-r--r-- 1 root   root    10907 May 28 18:47 ld.so.cache
-rw-r--r-- 1 root   root       34 Dec 16  2020 ld.so.conf
drwxr-xr-x 1 nobody nogroup  4096 Oct  4  2023 ld.so.conf.d
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:32 ldap
-rw-r--r-- 1 root   root      267 Oct 15  2021 legal
-rw-r--r-- 1 root   root      191 Mar 17  2022 libaudit.conf
-rw-r--r-- 1 root   root     2996 Sep 25  2023 locale.alias
-rw-r--r-- 1 root   root     9458 Nov 30  2023 locale.gen
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:32 logcheck
-rw-r--r-- 1 root   root    10734 Nov 11  2021 login.defs
-rw-r--r-- 1 root   root      592 May 25  2022 logrotate.conf
drwxr-xr-x 1 root   root     4096 May 17 20:43 logrotate.d
-rw-r--r-- 1 root   root      104 Aug  2  2023 lsb-release
-rw-r--r-- 1 root   root       33 Nov 30  2023 machine-id
-rw-r--r-- 1 root   root    72029 Mar 21  2022 mime.types
-rw-r--r-- 1 root   root      744 Jan  8  2022 mke2fs.conf
drwxr-xr-x 1 nobody nogroup  4096 Nov 30  2023 modprobe.d
-rw-r--r-- 1 root   root      195 Nov 30  2023 modules
drwxr-xr-x 1 nobody nogroup  4096 Nov 30  2023 modules-load.d
lrwxrwxrwx 1 root   root       19 Jun 12 23:44 mtab -> ../proc/self/mounts
-rw-r--r-- 1 root   root      767 Mar 24  2022 netconfig
-rw-r--r-- 1 root   root       91 Oct 15  2021 networks
-rw-r--r-- 1 root   root      494 Dec 16  2020 nsswitch.conf
drwxr-xr-x 1 nobody nogroup  4096 Oct  4  2023 opt
lrwxrwxrwx 1 root   root       21 Aug  2  2023 os-release -> ../usr/lib/os-release
-rw-r--r-- 1 root   root      552 Aug 12  2020 pam.conf
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 18:21 pam.d
-rw-r--r-- 1 root   root     1738 Jun 11 13:33 passwd
-rw-r--r-- 1 root   root     1735 Jun 11 13:33 passwd-
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:33 perl
-rw-r--r-- 1 root   root      582 Oct 15  2021 profile
drwxr-xr-x 1 nobody nogroup  4096 Oct  4  2023 profile.d
-rw-r--r-- 1 root   root     2932 Apr  1  2013 protocols
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 17:33 python3.10
drwxr-xr-x 1 root   root     4096 Jan  7 17:33 rc0.d
drwxr-xr-x 1 root   root     4096 Jan  7 17:33 rc1.d
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 18:21 rc2.d
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 18:21 rc3.d
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 18:21 rc4.d
drwxr-xr-x 1 nobody nogroup  4096 Jan  7 18:21 rc5.d
drwxr-xr-x 1 root   root     4096 Jan  7 17:33 rc6.d
drwxr-xr-x 1 root   root     4096 Jan  7 17:33 rcS.d
-rw-r--r-- 1 root   root       55 Jul  2 11:24 resolv.conf

Trying to change ownership:

# docker exec -it hmapp03  chown -v root:root  /etc/gss
chown: changing ownership of '/etc/gss': Operation not permitted
ffabreti commented 1 month ago

I have an older clone of the host that works ok. but the production host has the nobody:nogroup problem.

I've been testing with this dockerfile from nestybox github page

RESULT FROM OLDER CLONE VM:

# docker run -it --runtime=sysbox-runc --hostname hmapp03-orig --name hmapp03-orig --restart=unless-stopped ubuntu-jammy-systemd:v2

Welcome to Ubuntu 22.04.3 LTS!

[  OK  ] Created slice Slice /system/getty.
[  OK  ] Created slice Slice /system/modprobe.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[UNSUPP] Starting of Arbitrary Executable File Formats File System Automount Point unsupported.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Path Units.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slice Units.
[  OK  ] Reached target Swaps.
[  OK  ] Reached target Local Verity Protected Volumes.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Reached target Socket Units.
         Starting Journal Service...
         Starting Create List of Static Device Nodes...
         Starting Load Kernel Module configfs...
         Starting Load Kernel Module drm...
         Starting Load Kernel Module efi_pstore...
         Starting Load Kernel Module fuse...
         Starting Remount Root and Kernel File Systems...
         Starting Apply Kernel Variables...
         Starting Coldplug All udev Devices...
[  OK  ] Started Journal Service.
[  OK  ] Finished Create List of Static Device Nodes.
[  OK  ] Finished Load Kernel Module configfs.
[  OK  ] Finished Load Kernel Module drm.
[  OK  ] Finished Load Kernel Module efi_pstore.
[  OK  ] Finished Load Kernel Module fuse.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Starting Flush Journal to Persistent Storage...
         Starting Create System Users...
[  OK  ] Finished Flush Journal to Persistent Storage.
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Preparation for Local File Systems.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Create Volatile Files and Directories.
         Starting Network Name Resolution...
         Starting Record System Boot/Shutdown in UTMP...
[  OK  ] Finished Record System Boot/Shutdown in UTMP.
[  OK  ] Finished Coldplug All udev Devices.
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily apt download activities.
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Started Daily dpkg database backup timer.
[  OK  ] Started Periodic ext4 Online Metadata Check for All Filesystems.
[  OK  ] Started Message of the Day.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Basic System.
[  OK  ] Reached target Timer Units.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Started D-Bus System Message Bus.
         Starting Remove Stale Online ext4 Metadata Check Snapshots...
         Starting User Login Management...
         Starting Permit User Sessions...
[  OK  ] Finished Remove Stale Online ext4 Metadata Check Snapshots.
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Started User Login Management.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Record Runlevel Change in UTMP...
[  OK  ] Finished Record Runlevel Change in UTMP.

Ubuntu 22.04.3 LTS hmapp03-orig console

hmapp03-orig login: root
Password:
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.5.0-41-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@hmapp03-orig:~# ls -la /etc
total 436
drwxr-xr-x 1 root root    4096 Jul  9 18:06 .
drwxr-xr-x 1 root root    4096 Jul  9 18:06 ..
-rw------- 1 root root       0 Oct  4  2023 .pwd.lock
-rw-r--r-- 1 root root    3028 Oct  4  2023 adduser.conf
drwxr-xr-x 1 root root    4096 Nov 30  2023 alternatives
drwxr-xr-x 8 root root    4096 Oct  4  2023 apt
-rw-r--r-- 1 root root    2319 Jan  6  2022 bash.bashrc
-rw-r--r-- 1 root root     367 Dec 16  2020 bindresvport.blacklist
drwxr-xr-x 2 root root    4096 Sep 19  2023 binfmt.d
drwxr-xr-x 3 root root    4096 Nov 30  2023 ca-certificates
-rw-r--r-- 1 root root    5892 Nov 30  2023 ca-certificates.conf
drwxr-xr-x 2 root root    4096 Oct  4  2023 cloud
drwxr-xr-x 2 root root    4096 Oct  4  2023 cron.d
drwxr-xr-x 2 root root    4096 Oct  4  2023 cron.daily
drwxr-xr-x 4 root root    4096 Nov 30  2023 dbus-1
-rw-r--r-- 1 root root    2969 Feb 20  2022 debconf.conf
-rw-r--r-- 1 root root      13 Aug 22  2021 debian_version
drwxr-xr-x 1 root root    4096 Nov 30  2023 default
-rw-r--r-- 1 root root     604 Sep 15  2018 deluser.conf
drwxr-xr-x 2 root root    4096 Nov 30  2023 depmod.d
drwxr-xr-x 4 root root    4096 Oct  4  2023 dpkg
-rw-r--r-- 1 root root     685 Jan  8  2022 e2scrub.conf
[...]

RESULT FROM PRODUCTION VM:

# docker run -it --runtime=sysbox-runc --hostname hmapp03-orig --name hmapp03-orig --restart=unless-stopped ubuntu-jammy-systemd:v2

Welcome to Ubuntu 22.04.3 LTS!

[  OK  ] Created slice Slice /system/getty.
[  OK  ] Created slice Slice /system/modprobe.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[UNSUPP] Starting of Arbitrary Executable File Formats File System Automount Point unsupported.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Path Units.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slice Units.
[  OK  ] Reached target Swaps.
[  OK  ] Reached target Local Verity Protected Volumes.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Reached target Socket Units.
         Starting Journal Service...
         Starting Create List of Static Device Nodes...
         Starting Load Kernel Module configfs...
         Starting Load Kernel Module drm...
         Starting Load Kernel Module efi_pstore...
         Starting Load Kernel Module fuse...
         Starting Remount Root and Kernel File Systems...
         Starting Apply Kernel Variables...
         Starting Coldplug All udev Devices...
[  OK  ] Started Journal Service.
[  OK  ] Finished Create List of Static Device Nodes.
[  OK  ] Finished Load Kernel Module configfs.
[  OK  ] Finished Load Kernel Module drm.
[  OK  ] Finished Load Kernel Module efi_pstore.
[  OK  ] Finished Load Kernel Module fuse.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Starting Flush Journal to Persistent Storage...
         Starting Create System Users...
[  OK  ] Finished Flush Journal to Persistent Storage.
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Preparation for Local File Systems.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Create Volatile Files and Directories.
         Starting Network Name Resolution...
         Starting Record System Boot/Shutdown in UTMP...
[  OK  ] Finished Coldplug All udev Devices.
[  OK  ] Finished Record System Boot/Shutdown in UTMP.
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily apt download activities.
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Started Daily dpkg database backup timer.
[  OK  ] Started Periodic ext4 Online Metadata Check for All Filesystems.
[  OK  ] Started Message of the Day.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Basic System.
[  OK  ] Reached target Timer Units.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Started D-Bus System Message Bus.
         Starting Remove Stale Online ext4 Metadata Check Snapshots...
         Starting User Login Management...
         Starting Permit User Sessions...
[  OK  ] Finished Remove Stale Online ext4 Metadata Check Snapshots.
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started Console Getty.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Started User Login Management.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Record Runlevel Change in UTMP...
[  OK  ] Finished Record Runlevel Change in UTMP.

Ubuntu 22.04.3 LTS hmapp03-orig console

hmapp03-orig login: root
Password:
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.5.0-41-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@hmapp03-orig:~# ls -la /etc
total 436
drwxr-xr-x 1 root   root     4096 Jul  9 18:12 .
drwxr-xr-x 1 root   root     4096 Jul  9 18:12 ..
-rw------- 1 root   root        0 Oct  4  2023 .pwd.lock
-rw-r--r-- 1 root   root     3028 Oct  4  2023 adduser.conf
drwxr-xr-x 1 root   root     4096 Nov 30  2023 alternatives
drwxr-xr-x 8 root   root     4096 Oct  4  2023 apt
-rw-r--r-- 1 root   root     2319 Jan  6  2022 bash.bashrc
-rw-r--r-- 1 root   root      367 Dec 16  2020 bindresvport.blacklist
drwxr-xr-x 2 root   root     4096 Sep 19  2023 binfmt.d
drwxr-xr-x 3 root   root     4096 Nov 30  2023 ca-certificates
-rw-r--r-- 1 root   root     5892 Nov 30  2023 ca-certificates.conf
drwxr-xr-x 2 nobody nogroup  4096 Oct  4  2023 cloud
drwxr-xr-x 2 root   root     4096 Oct  4  2023 cron.d
drwxr-xr-x 2 root   root     4096 Oct  4  2023 cron.daily
drwxr-xr-x 4 root   root     4096 Nov 30  2023 dbus-1
-rw-r--r-- 1 root   root     2969 Feb 20  2022 debconf.conf
-rw-r--r-- 1 root   root       13 Aug 22  2021 debian_version
drwxr-xr-x 1 root   root     4096 Nov 30  2023 default
-rw-r--r-- 1 root   root      604 Sep 15  2018 deluser.conf
drwxr-xr-x 2 nobody nogroup  4096 Nov 30  2023 depmod.d
drwxr-xr-x 4 nobody nogroup  4096 Oct  4  2023 dpkg
-rw-r--r-- 1 root   root      685 Jan  8  2022 e2scrub.conf
ctalledo commented 1 month ago

Hi @ffabreti, thanks for using Sysbox and filing the issue.

I have upgraded my host Ubuntu 22.04.3 from kernel 5.15.0-113 to 6.5.0-41

Good, Sysbox works better with kernel 5.19+ (which supports ID-mapped-mounts and overlayfs on top of them).

I'm looking at LVM now, because my host /var/lib/docker is mounted on a LVM volume

That shouldn't be an issue, since the LVM is ext4 and ID-mapped-mounts work fine with ext4.

I've noticed on hmapp03, var/lib/docker is not idmapped!

That is a problem: if the kernel is 5.12+, and you mount a host dir into the Sysbox container's /var/lib/docker, Sysbox should have ID-mapped that host dir (within the mount namespace of the container).

How does findmnt look inside the Sysbox container (just plain findmnt, not findmnt -J)?

ffabreti commented 1 month ago

Thanks @ctalledo,

Just explaining that since I've posted this issue I have moved host /var/lib/docker from a LVM to a plain partition (sdc1). But as you pointed out, it is not a problem:

# df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   39G   18G   20G  48% /
/dev/sdc1                          147G   46G   94G  33% /var/lib/docker
# mount
/dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (rw,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
/dev/sdc1 on /var/lib/docker type ext4 (rw,relatime)

Another thing that comes to mind is that I am using a docker volume for the varlibdocker of the sysbox container (not a bind mount) so my docker run is:

docker run \
--runtime=sysbox-runc \
--hostname hmapp03 \
--name hmapp03 \
--restart=unless-stopped \
--mount source=varlibdocker-hmapp03-v1,target=/var/lib/docker \
--ip 192.168.61.5 \
--network br-hmnet \
-p 2211:22 \
-p 5000-5050:5000-5050 \
--detach \
ubuntu-jammy-systemd-docker:v4 <this is a custom image based on a nestybox provided image>

#in the host:
# ls -la /var/lib/docker/volumes
total 168
drwx-----x 15 root root   4096 Jul  5 18:23 .
drwx--x--- 14 root root   4096 Jul  4 18:31 ..
brw-------  1 root root  8, 33 Jul  4 18:31 backingFsBlockDev
-rw-------  1 root root 131072 Jul  5 18:23 metadata.db
drwx-----x  3 root root   4096 May 17 15:44 varlibdocker-hmapp03-v1

findmnt inside sysbox container:

# docker exec -it hmapp03 bash
root@hmapp03:/# findmnt | cat
TARGET                                                                                               SOURCE                                                                                                                           FSTYPE   OPTIONS
/                                                                                                    overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/YKFESOSMHM6Z6CQID7P4P4H3DN:/var/lib/docker/overlay2/l/DAEVDBOZUPJINUQRLNVL2AXQRZ:/var/lib/docker/overlay2/l/2Y25TM5F7W3MDXAFE4RWSXYBYA:/var/lib/docker/overlay2/l/VQJ6BA3564D65XP2YOLXMM2XPO:/var/lib/docker/overlay2/l/AMSLSHZXUEZRUVE6S76C7ITFXK:/var/lib/docker/overlay2/l/MTD6J762Q4K6XBMIDH65CT55Z3:/var/lib/docker/overlay2/l/32OYFFIY5KGNXMKSYJAG636LDT:/var/lib/docker/overlay2/l/D3BINRCBRXO2PNY3WVUIWPYX2U:/var/lib/docker/overlay2/l/5FNLNOK5VPY3C7HI5HGCMO4B2J:/var/lib/docker/overlay2/l/PJX37BIXPOV7QULXJFHIWIEB2E:/var/lib/docker/overlay2/l/FOCPSCVXCLWATQBHZCPOD6H2RN:/var/lib/docker/overlay2/l/5IWJDNKDG64HJFOPLRNRYERE4E,upperdir=/var/lib/docker/overlay2/87d1553acd6027d1b0a47d459dd85f5e52ed71cfa6dea415640acee49872bcbe/diff,workdir=/var/lib/docker/overlay2/87d1553acd6027d1b0a47d459dd85f5e52ed71cfa6dea415640acee49872bcbe/work,nouserxattr
|-/sys                                                                                               sysfs                                                                                                                            sysfs    rw,nosuid,nodev,noexec,relatime
| |-/sys/firmware                                                                                    tmpfs                                                                                                                            tmpfs    ro,relatime,uid=165536,gid=165536,inode64
| |-/sys/fs/cgroup                                                                                   cgroup                                                                                                                           cgroup2  rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
| |-/sys/devices/virtual                                                                             sysboxfs[/sys/devices/virtual]                                                                                                   fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| |-/sys/kernel                                                                                      sysboxfs[/sys/kernel]                                                                                                            fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| `-/sys/module/nf_conntrack/parameters                                                              sysboxfs[/sys/module/nf_conntrack/parameters]                                                                                    fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
|-/proc                                                                                              proc                                                                                                                             proc     rw,nosuid,nodev,noexec,relatime
| |-/proc/bus                                                                                        proc[/bus]                                                                                                                       proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/fs                                                                                         proc[/fs]                                                                                                                        proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/irq                                                                                        proc[/irq]                                                                                                                       proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/sysrq-trigger                                                                              proc[/sysrq-trigger]                                                                                                             proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/acpi                                                                                       tmpfs                                                                                                                            tmpfs    ro,relatime,uid=165536,gid=165536,inode64
| |-/proc/keys                                                                                       udev[/null]                                                                                                                      devtmpfs rw,nosuid,relatime,size=8118748k,nr_inodes=2029687,mode=755,inode64
| |-/proc/timer_list                                                                                 udev[/null]                                                                                                                      devtmpfs rw,nosuid,relatime,size=8118748k,nr_inodes=2029687,mode=755,inode64
| |-/proc/scsi                                                                                       tmpfs                                                                                                                            tmpfs    ro,relatime,uid=165536,gid=165536,inode64
| |-/proc/swaps                                                                                      sysboxfs[/proc/swaps]                                                                                                            fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| |-/proc/sys                                                                                        sysboxfs[/proc/sys]                                                                                                              fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| `-/proc/uptime                                                                                     sysboxfs[/proc/uptime]                                                                                                           fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
|-/dev                                                                                               tmpfs                                                                                                                            tmpfs    rw,nosuid,size=65536k,mode=755,uid=165536,gid=165536,inode64
| |-/dev/mqueue                                                                                      mqueue                                                                                                                           mqueue   rw,nosuid,nodev,noexec,relatime
| |-/dev/pts                                                                                         devpts                                                                                                                           devpts   rw,nosuid,noexec,relatime,gid=165541,mode=620,ptmxmode=666
| |-/dev/shm                                                                                         shm                                                                                                                              tmpfs    rw,nosuid,nodev,noexec,relatime,size=65536k,uid=165536,gid=165536,inode64
| |-/dev/null                                                                                        udev[/null]                                                                                                                      devtmpfs rw,nosuid,relatime,size=8118748k,nr_inodes=2029687,mode=755,inode64
| |-/dev/random                                                                                      udev[/random]                                                                                                                    devtmpfs rw,nosuid,relatime,size=8118748k,nr_inodes=2029687,mode=755,inode64
| |-/dev/kmsg                                                                                        udev[/null]                                                                                                                      devtmpfs rw,nosuid,relatime,size=8118748k,nr_inodes=2029687,mode=755,inode64
| |-/dev/full                                                                                        udev[/full]                                                                                                                      devtmpfs rw,nosuid,relatime,size=8118748k,nr_inodes=2029687,mode=755,inode64
| |-/dev/tty                                                                                         udev[/tty]                                                                                                                       devtmpfs rw,nosuid,relatime,size=8118748k,nr_inodes=2029687,mode=755,inode64
| |-/dev/zero                                                                                        udev[/zero]                                                                                                                      devtmpfs rw,nosuid,relatime,size=8118748k,nr_inodes=2029687,mode=755,inode64
| `-/dev/urandom                                                                                     udev[/urandom]                                                                                                                   devtmpfs rw,nosuid,relatime,size=8118748k,nr_inodes=2029687,mode=755,inode64
|-/run                                                                                               tmpfs                                                                                                                            tmpfs    rw,nosuid,nodev,relatime,size=65536k,mode=755,uid=165536,gid=165536,inode64
| |-/run/lock                                                                                        tmpfs                                                                                                                            tmpfs    rw,nosuid,nodev,noexec,relatime,size=4096k,uid=165536,gid=165536,inode64
| |-/run/docker/netns/2c52d092a111                                                                   nsfs[net:[4026533153]]                                                                                                           nsfs     rw
| |-/run/docker/netns/c2a14cdaaee4                                                                   nsfs[net:[4026532980]]                                                                                                           nsfs     rw
| |-/run/docker/netns/df2968aa720d                                                                   nsfs[net:[4026533024]]                                                                                                           nsfs     rw
| |-/run/docker/netns/8df3fae7d81b                                                                   nsfs[net:[4026533067]]                                                                                                           nsfs     rw
| `-/run/docker/netns/89e5289a9dbd                                                                   nsfs[net:[4026533111]]                                                                                                           nsfs     rw
|-/var/lib/docker                                                                                    /dev/sdc1[/volumes/varlibdocker-hmapp03-v1/_data]                                                                                ext4     rw,relatime
| |-/var/lib/docker/overlay2/38b865df903ac9438b0ecf725ccae2d5f0b634bf92e84f3a1f1a6b5a5e9c215a/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/URLKFBUDMI7JDB7CREHPL74SGL:/var/lib/docker/overlay2/l/SLEZCTYO4BKMPG2DJ6NVT7JAJV:/var/lib/docker/overlay2/l/XGL3CH57WPKBZCWDMVHESLV5YI:/var/lib/docker/overlay2/l/TZKT3PDY7SWMPJ5QH624SOFBQK:/var/lib/docker/overlay2/l/QU2IHZQFV2EJTXFPXC6Y62NC4T:/var/lib/docker/overlay2/l/LFF774KAFIVLJKYC5GXSJNTHTQ:/var/lib/docker/overlay2/l/E6RKL5PF7DWM4F2LVWGAUWUWBM:/var/lib/docker/overlay2/l/ALE7EENLP23LDZ3243ACOOIQVK:/var/lib/docker/overlay2/l/QAYYGH3NPYTPXWGHLYOKTN4PJC:/var/lib/docker/overlay2/l/WUNCAHTG2BG5JDTPVB6PULWFNW:/var/lib/docker/overlay2/l/4HPT25MP4B35EVOOFAKGYWMO7K:/var/lib/docker/overlay2/l/WWR7422SH6JAWJEX7HKO65TRC5:/var/lib/docker/overlay2/l/OOIW73CPUJOXE4FVK3VBNTESBC:/var/lib/docker/overlay2/l/KCLC4WRNYZPZAFHL5UNP3DPRQI:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/38b865df903ac9438b0ecf725ccae2d5f0b634bf92e84f3a1f1a6b5a5e9c215a/diff,workdir=/var/lib/docker/overlay2/38b865df903ac9438b0ecf725ccae2d5f0b634bf92e84f3a1f1a6b5a5e9c215a/work,redirect_dir=nofollow,userxattr
| |-/var/lib/docker/overlay2/1ddd51420c29089e92d54de2571ec0605649984afb430d8e3b2b09a6c5c287ac/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ID2UKGKSDL5WDVM2B4YONXNGKX:/var/lib/docker/overlay2/l/TDRXF65OY35S2NTMLNDZS3PH7M:/var/lib/docker/overlay2/l/LTABWD2NVVPEP3Q4UCYNTTLWVL:/var/lib/docker/overlay2/l/ALE3GTUMJLMIIPKOEXPBJ4FQC2:/var/lib/docker/overlay2/l/K6BZ3BNKYZGHAVNL2U6ZLHIWCX:/var/lib/docker/overlay2/l/BGVKLVICTJBYLO3TBC7TVHWNLJ:/var/lib/docker/overlay2/l/UZ7VSFENG2I3UUTFH7TOYLSCVV:/var/lib/docker/overlay2/l/LGZC4JAIDMVGCIV7ZZOQE2BFWI:/var/lib/docker/overlay2/l/PMRLEB4RWVVVQSRBAHOZ5MSWXP:/var/lib/docker/overlay2/l/BHTKFKFS5JAWTMNJRYKFF6CF6U:/var/lib/docker/overlay2/l/4HPT25MP4B35EVOOFAKGYWMO7K:/var/lib/docker/overlay2/l/WWR7422SH6JAWJEX7HKO65TRC5:/var/lib/docker/overlay2/l/OOIW73CPUJOXE4FVK3VBNTESBC:/var/lib/docker/overlay2/l/KCLC4WRNYZPZAFHL5UNP3DPRQI:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/1ddd51420c29089e92d54de2571ec0605649984afb430d8e3b2b09a6c5c287ac/diff,workdir=/var/lib/docker/overlay2/1ddd51420c29089e92d54de2571ec0605649984afb430d8e3b2b09a6c5c287ac/work,redirect_dir=nofollow,userxattr
| |-/var/lib/docker/overlay2/d5fd1d3c15285fe84fd3d322b83d100d5e75517031b7b567be109c61b1ceb9a7/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ZL2KKYSKGZYLW5O3Z73BV5FYXR:/var/lib/docker/overlay2/l/K6DPLAIARVXWO2RORDPMZ7TE5G:/var/lib/docker/overlay2/l/M55LXWYNV6QMEQ5HN4TYGWB3B4:/var/lib/docker/overlay2/l/MVZJ4F2ACJM6HOO34FOJUEJQE5:/var/lib/docker/overlay2/l/25JJQECF7BLUIFANCRI6E3OVCY:/var/lib/docker/overlay2/l/4TYXPERPMEXUNBM6DSDULSWJJF:/var/lib/docker/overlay2/l/DRG5F6Z6OMSTULCUY3R7W7A2I6:/var/lib/docker/overlay2/l/TEL5HDTBHMHXWQIP4U2RCHMS22:/var/lib/docker/overlay2/l/P4L3P27ORLHUBFVMJVBGPZUEST:/var/lib/docker/overlay2/l/3ZZMO7XU6UJESPUIYZI2WDU5W3:/var/lib/docker/overlay2/l/KTAVS67FJALBGNFQTSN4HFKVRJ:/var/lib/docker/overlay2/l/SCDBG2QVH75IQET75ADJJUQ7Y5:/var/lib/docker/overlay2/l/V4VEZI5ANZOEPDR57VM52JEJWA:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/d5fd1d3c15285fe84fd3d322b83d100d5e75517031b7b567be109c61b1ceb9a7/diff,workdir=/var/lib/docker/overlay2/d5fd1d3c15285fe84fd3d322b83d100d5e75517031b7b567be109c61b1ceb9a7/work,redirect_dir=nofollow,userxattr
| |-/var/lib/docker/overlay2/94d3936d0a4b183e106eeb52d607a2d85298c037bf390b1196681a8bbda459a0/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/PUP3NOSWRKERKDSQKWWAPOFJNR:/var/lib/docker/overlay2/l/OSF6M6FMVBV3IJ7FSH4LI5N3F4:/var/lib/docker/overlay2/l/T46UFOCIPHOFTZ6EQCXSDWCWOE:/var/lib/docker/overlay2/l/VKX3DK3HK5PJZFJEV7UWOA5RU6:/var/lib/docker/overlay2/l/2RWLBB4BCC22KPGHUVVZ6LMKRR:/var/lib/docker/overlay2/l/NX65NGDUXIDDHOE762BW445ZYU:/var/lib/docker/overlay2/l/FOAID5ET25UO66PW5BNPI3IUVP:/var/lib/docker/overlay2/l/OHJJILK5VX7QOPLSHVLUQD7J5H:/var/lib/docker/overlay2/l/7Z3HQRZ6GRE6AFKCJGGWXNTOPG:/var/lib/docker/overlay2/l/NKMEUSMYQ33RS5V2WGTGQHF37M:/var/lib/docker/overlay2/l/3ZZMO7XU6UJESPUIYZI2WDU5W3:/var/lib/docker/overlay2/l/KTAVS67FJALBGNFQTSN4HFKVRJ:/var/lib/docker/overlay2/l/SCDBG2QVH75IQET75ADJJUQ7Y5:/var/lib/docker/overlay2/l/V4VEZI5ANZOEPDR57VM52JEJWA:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/94d3936d0a4b183e106eeb52d607a2d85298c037bf390b1196681a8bbda459a0/diff,workdir=/var/lib/docker/overlay2/94d3936d0a4b183e106eeb52d607a2d85298c037bf390b1196681a8bbda459a0/work,redirect_dir=nofollow,userxattr
| `-/var/lib/docker/overlay2/06ed7e3c8b72c21557fc1b0de430d03ef5f5dec0ddf96baa160fc50fe6dcb08b/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SWQBQKDUOHWKBHKZFI53TVXFHF:/var/lib/docker/overlay2/l/JMLUD7ZMFMEC6GB7W2SL6ZGBPM:/var/lib/docker/overlay2/l/QLOX7MTGZVLC3WCVRM2O65WE6T:/var/lib/docker/overlay2/l/QXBMMDMDAXDHPXCR6Q5IAZCACC:/var/lib/docker/overlay2/l/RMEA2WRCJB5BFMUMV65I6FP7D5:/var/lib/docker/overlay2/l/GI65S43RQ7OQ34S4XT3YK6DH5T:/var/lib/docker/overlay2/l/J32RM357H3JNJU3PQATZRILPB3:/var/lib/docker/overlay2/l/F3DZTXATKSVZRDU7TOE25SAOAV:/var/lib/docker/overlay2/l/ITOLGDS3JIP7DDRCLC43W6DLOT:/var/lib/docker/overlay2/l/JMVQZOGQAZYBAAWOYRXIISX4V6:/var/lib/docker/overlay2/l/TXRBEOBOSX2UZHPASF3IPLOCVX,upperdir=/var/lib/docker/overlay2/06ed7e3c8b72c21557fc1b0de430d03ef5f5dec0ddf96baa160fc50fe6dcb08b/diff,workdir=/var/lib/docker/overlay2/06ed7e3c8b72c21557fc1b0de430d03ef5f5dec0ddf96baa160fc50fe6dcb08b/work,redirect_dir=nofollow,userxattr
|-/etc/resolv.conf                                                                                   /dev/sdc1[/containers/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa/resolv.conf]                              ext4     rw,relatime,idmapped
|-/etc/hostname                                                                                      /dev/sdc1[/containers/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa/hostname]                                 ext4     rw,relatime,idmapped
|-/etc/hosts                                                                                         /dev/sdc1[/containers/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa/hosts]                                    ext4     rw,relatime,idmapped
|-/var/lib/buildkit                                                                                  /dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/buildkit/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]     ext4     rw,relatime,idmapped
|-/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs                                         /dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/containerd/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]   ext4     rw,relatime,idmapped
|-/var/lib/rancher/k3s                                                                               /dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/rancher-k3s/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]  ext4     rw,relatime,idmapped
|-/var/lib/rancher/rke2                                                                              /dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/rancher-rke2/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa] ext4     rw,relatime,idmapped
|-/var/lib/kubelet                                                                                   /dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/kubelet/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]      ext4     rw,relatime,idmapped
|-/var/lib/k0s                                                                                       /dev/mapper/ubuntu--vg-ubuntu--lv[/var/lib/sysbox/k0s/6ac160a4b546bfcca6d326402adeba578174fe0cb1391575f927bc4c257a15fa]          ext4     rw,relatime,idmapped
|-/usr/src                                                                                           /dev/mapper/ubuntu--vg-ubuntu--lv[/usr/src]                                                                                      ext4     ro,relatime,idmapped
| `-/usr/src/linux-headers-6.5.0-41-generic                                                          /dev/mapper/ubuntu--vg-ubuntu--lv[/usr/src/linux-headers-6.5.0-41-generic]                                                       ext4     ro,relatime,idmapped
`-/usr/lib/modules/6.5.0-41-generic                                                                  /dev/mapper/ubuntu--vg-ubuntu--lv[/usr/lib/modules/6.5.0-41-generic]                                                             ext4     ro,relatime,idmapped
ffabreti commented 1 month ago

Although I don't quite understand docker overlays, I've been trying to find something that explains the problem inside overlay2 directory. I wish I had any means to remedy the permissions, because it is a production system and I'm sure I'll have problems. attached follows a screenshot of overlay2 dir on host and a container listing of the same dir. Maybe it rings any bells... image

I also have upgrade sysbox on the host to 0.6.4, no changes seen.

ctalledo commented 1 month ago

Hi @ffabreti,

Thanks for the extra info.

Another thing that comes to mind is that I am using a docker volume for the varlibdocker of the sysbox container (not a bind mount)

Got it.

findmnt inside sysbox container: ...

|-/var/lib/docker                                                                                    /dev/sdc1[/volumes/varlibdocker-hmapp03-v1/_data]                                                                                ext4     rw,relatime
| |-/var/lib/docker/overlay2/38b865df903ac9438b0ecf725ccae2d5f0b634bf92e84f3a1f1a6b5a5e9c215a/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/URLKFBUDMI7JDB7CREHPL74SGL:/var/lib/docker/overlay2/l/SLEZCTYO4BKMPG2DJ6NVT7JAJV:/var/lib/docker/overlay2/l/XGL3CH57WPKBZCWDMVHESLV5YI:/var/lib/docker/overlay2/l/TZKT3PDY7SWMPJ5QH624SOFBQK:/var/lib/docker/overlay2/l/QU2IHZQFV2EJTXFPXC6Y62NC4T:/var/lib/docker/overlay2/l/LFF774KAFIVLJKYC5GXSJNTHTQ:/var/lib/docker/overlay2/l/E6RKL5PF7DWM4F2LVWGAUWUWBM:/var/lib/docker/overlay2/l/ALE7EENLP23LDZ3243ACOOIQVK:/var/lib/docker/overlay2/l/QAYYGH3NPYTPXWGHLYOKTN4PJC:/var/lib/docker/overlay2/l/WUNCAHTG2BG5JDTPVB6PULWFNW:/var/lib/docker/overlay2/l/4HPT25MP4B35EVOOFAKGYWMO7K:/var/lib/docker/overlay2/l/WWR7422SH6JAWJEX7HKO65TRC5:/var/lib/docker/overlay2/l/OOIW73CPUJOXE4FVK3VBNTESBC:/var/lib/docker/overlay2/l/KCLC4WRNYZPZAFHL5UNP3DPRQI:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/38b865df903ac9438b0ecf725ccae2d5f0b634bf92e84f3a1f1a6b5a5e9c215a/diff,workdir=/var/lib/docker/overlay2/38b865df903ac9438b0ecf725ccae2d5f0b634bf92e84f3a1f1a6b5a5e9c215a/work,redirect_dir=nofollow,userxattr
| |-/var/lib/docker/overlay2/1ddd51420c29089e92d54de2571ec0605649984afb430d8e3b2b09a6c5c287ac/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ID2UKGKSDL5WDVM2B4YONXNGKX:/var/lib/docker/overlay2/l/TDRXF65OY35S2NTMLNDZS3PH7M:/var/lib/docker/overlay2/l/LTABWD2NVVPEP3Q4UCYNTTLWVL:/var/lib/docker/overlay2/l/ALE3GTUMJLMIIPKOEXPBJ4FQC2:/var/lib/docker/overlay2/l/K6BZ3BNKYZGHAVNL2U6ZLHIWCX:/var/lib/docker/overlay2/l/BGVKLVICTJBYLO3TBC7TVHWNLJ:/var/lib/docker/overlay2/l/UZ7VSFENG2I3UUTFH7TOYLSCVV:/var/lib/docker/overlay2/l/LGZC4JAIDMVGCIV7ZZOQE2BFWI:/var/lib/docker/overlay2/l/PMRLEB4RWVVVQSRBAHOZ5MSWXP:/var/lib/docker/overlay2/l/BHTKFKFS5JAWTMNJRYKFF6CF6U:/var/lib/docker/overlay2/l/4HPT25MP4B35EVOOFAKGYWMO7K:/var/lib/docker/overlay2/l/WWR7422SH6JAWJEX7HKO65TRC5:/var/lib/docker/overlay2/l/OOIW73CPUJOXE4FVK3VBNTESBC:/var/lib/docker/overlay2/l/KCLC4WRNYZPZAFHL5UNP3DPRQI:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/1ddd51420c29089e92d54de2571ec0605649984afb430d8e3b2b09a6c5c287ac/diff,workdir=/var/lib/docker/overlay2/1ddd51420c29089e92d54de2571ec0605649984afb430d8e3b2b09a6c5c287ac/work,redirect_dir=nofollow,userxattr
| |-/var/lib/docker/overlay2/d5fd1d3c15285fe84fd3d322b83d100d5e75517031b7b567be109c61b1ceb9a7/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ZL2KKYSKGZYLW5O3Z73BV5FYXR:/var/lib/docker/overlay2/l/K6DPLAIARVXWO2RORDPMZ7TE5G:/var/lib/docker/overlay2/l/M55LXWYNV6QMEQ5HN4TYGWB3B4:/var/lib/docker/overlay2/l/MVZJ4F2ACJM6HOO34FOJUEJQE5:/var/lib/docker/overlay2/l/25JJQECF7BLUIFANCRI6E3OVCY:/var/lib/docker/overlay2/l/4TYXPERPMEXUNBM6DSDULSWJJF:/var/lib/docker/overlay2/l/DRG5F6Z6OMSTULCUY3R7W7A2I6:/var/lib/docker/overlay2/l/TEL5HDTBHMHXWQIP4U2RCHMS22:/var/lib/docker/overlay2/l/P4L3P27ORLHUBFVMJVBGPZUEST:/var/lib/docker/overlay2/l/3ZZMO7XU6UJESPUIYZI2WDU5W3:/var/lib/docker/overlay2/l/KTAVS67FJALBGNFQTSN4HFKVRJ:/var/lib/docker/overlay2/l/SCDBG2QVH75IQET75ADJJUQ7Y5:/var/lib/docker/overlay2/l/V4VEZI5ANZOEPDR57VM52JEJWA:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/d5fd1d3c15285fe84fd3d322b83d100d5e75517031b7b567be109c61b1ceb9a7/diff,workdir=/var/lib/docker/overlay2/d5fd1d3c15285fe84fd3d322b83d100d5e75517031b7b567be109c61b1ceb9a7/work,redirect_dir=nofollow,userxattr
| |-/var/lib/docker/overlay2/94d3936d0a4b183e106eeb52d607a2d85298c037bf390b1196681a8bbda459a0/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/PUP3NOSWRKERKDSQKWWAPOFJNR:/var/lib/docker/overlay2/l/OSF6M6FMVBV3IJ7FSH4LI5N3F4:/var/lib/docker/overlay2/l/T46UFOCIPHOFTZ6EQCXSDWCWOE:/var/lib/docker/overlay2/l/VKX3DK3HK5PJZFJEV7UWOA5RU6:/var/lib/docker/overlay2/l/2RWLBB4BCC22KPGHUVVZ6LMKRR:/var/lib/docker/overlay2/l/NX65NGDUXIDDHOE762BW445ZYU:/var/lib/docker/overlay2/l/FOAID5ET25UO66PW5BNPI3IUVP:/var/lib/docker/overlay2/l/OHJJILK5VX7QOPLSHVLUQD7J5H:/var/lib/docker/overlay2/l/7Z3HQRZ6GRE6AFKCJGGWXNTOPG:/var/lib/docker/overlay2/l/NKMEUSMYQ33RS5V2WGTGQHF37M:/var/lib/docker/overlay2/l/3ZZMO7XU6UJESPUIYZI2WDU5W3:/var/lib/docker/overlay2/l/KTAVS67FJALBGNFQTSN4HFKVRJ:/var/lib/docker/overlay2/l/SCDBG2QVH75IQET75ADJJUQ7Y5:/var/lib/docker/overlay2/l/V4VEZI5ANZOEPDR57VM52JEJWA:/var/lib/docker/overlay2/l/ZVFK4KDKTQRMJIZBFJHB3CKZRK,upperdir=/var/lib/docker/overlay2/94d3936d0a4b183e106eeb52d607a2d85298c037bf390b1196681a8bbda459a0/diff,workdir=/var/lib/docker/overlay2/94d3936d0a4b183e106eeb52d607a2d85298c037bf390b1196681a8bbda459a0/work,redirect_dir=nofollow,userxattr
| `-/var/lib/docker/overlay2/06ed7e3c8b72c21557fc1b0de430d03ef5f5dec0ddf96baa160fc50fe6dcb08b/merged overlay                                                                                                                          overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SWQBQKDUOHWKBHKZFI53TVXFHF:/var/lib/docker/overlay2/l/JMLUD7ZMFMEC6GB7W2SL6ZGBPM:/var/lib/docker/overlay2/l/QLOX7MTGZVLC3WCVRM2O65WE6T:/var/lib/docker/overlay2/l/QXBMMDMDAXDHPXCR6Q5IAZCACC:/var/lib/docker/overlay2/l/RMEA2WRCJB5BFMUMV65I6FP7D5:/var/lib/docker/overlay2/l/GI65S43RQ7OQ34S4XT3YK6DH5T:/var/lib/docker/overlay2/l/J32RM357H3JNJU3PQATZRILPB3:/var/lib/docker/overlay2/l/F3DZTXATKSVZRDU7TOE25SAOAV:/var/lib/docker/overlay2/l/ITOLGDS3JIP7DDRCLC43W6DLOT:/var/lib/docker/overlay2/l/JMVQZOGQAZYBAAWOYRXIISX4V6:/var/lib/docker/overlay2/l/TXRBEOBOSX2UZHPASF3IPLOCVX,upperdir=/var/lib/docker/overlay2/06ed7e3c8b72c21557fc1b0de430d03ef5f5dec0ddf96baa160fc50fe6dcb08b/diff,workdir=/var/lib/docker/overlay2/06ed7e3c8b72c21557fc1b0de430d03ef5f5dec0ddf96baa160fc50fe6dcb08b/work,redirect_dir=nofollow,userxattr

That looks fine, assuming the submounts under the container's /var/lib/docker are from Docker containers running inside the Sysbox container.

In any case, the problem of files showing up with nobody:nogroup under the Sysbox container's /etc should not be related to the mount on /var/lib/docker.

Regarding the files under /etc:

root@hmapp03-orig:~# ls -la /etc
total 436
drwxr-xr-x 1 root   root     4096 Jul  9 18:12 .
drwxr-xr-x 1 root   root     4096 Jul  9 18:12 ..
-rw------- 1 root   root        0 Oct  4  2023 .pwd.lock
-rw-r--r-- 1 root   root     3028 Oct  4  2023 adduser.conf
drwxr-xr-x 1 root   root     4096 Nov 30  2023 alternatives
drwxr-xr-x 8 root   root     4096 Oct  4  2023 apt
-rw-r--r-- 1 root   root     2319 Jan  6  2022 bash.bashrc
-rw-r--r-- 1 root   root      367 Dec 16  2020 bindresvport.blacklist
drwxr-xr-x 2 root   root     4096 Sep 19  2023 binfmt.d
drwxr-xr-x 3 root   root     4096 Nov 30  2023 ca-certificates
-rw-r--r-- 1 root   root     5892 Nov 30  2023 ca-certificates.conf
drwxr-xr-x 2 nobody nogroup  4096 Oct  4  2023 cloud
drwxr-xr-x 2 root   root     4096 Oct  4  2023 cron.d
drwxr-xr-x 2 root   root     4096 Oct  4  2023 cron.daily
drwxr-xr-x 4 root   root     4096 Nov 30  2023 dbus-1
-rw-r--r-- 1 root   root     2969 Feb 20  2022 debconf.conf
-rw-r--r-- 1 root   root       13 Aug 22  2021 debian_version
drwxr-xr-x 1 root   root     4096 Nov 30  2023 default
-rw-r--r-- 1 root   root      604 Sep 15  2018 deluser.conf
drwxr-xr-x 2 nobody nogroup  4096 Nov 30  2023 depmod.d
drwxr-xr-x 4 nobody nogroup  4096 Oct  4  2023 dpkg
-rw-r--r-- 1 root   root      685 Jan  8  2022 e2scrub.conf

So files cloud, depmod.d, and dpkg have nobody:nogroup ownership somehow.

From the findmnt output, I can see those files are not mounted but are instead part of the container image.

Do you know what created those files inside the container image?

Also, how do the files under the container's /etc show up when running the vanilla nestybox/ubuntu-jammy-systemd-docker image for example?

ffabreti commented 1 month ago

I have good news.

Also, how do the files under the container's /etc show up when running the vanilla nestybox/ubuntu-jammy-systemd-docker image for example?

I have tested with nestybox/ubuntu-jammy-systemd-docker image and I've noticed that back then (when I built the image) I had to make some changes, so I've build this v4 image, this is the diff between vanilla and v4:

# diff --color Dockerfile.vanilla Dockerfile.v4
7,9c7,9
< # This will run systemd and prompt for a user login; the default
< # user/password in this image is "admin/admin". Once you log in you
< # can run Docker inside as usual. You can also ssh into the image:
---
> # Systemd will show prompt for login; default user/pass 
> # is "root/my-great-pass". Once you log in you can run Docker inside as usual.
> # You can also ssh into the image:
14a15,19
> # history
> # - v1: nestybox original
> # - v2: downgrade docker to 23.0.6 because of a bug running --network=host when DIND
> # - v3: removed admin user, using root instead.
> # - v4: added convenience configs
16c21
< FROM ghcr.io/nestybox/ubuntu-jammy-systemd:latest
---
> FROM ubuntu-jammy-systemd:v3
18,19c23,25
< # Install Docker
< RUN apt-get update && apt-get install -y curl \
---
>
> # Install Docker and utils install
> RUN apt-get update && apt-get install -y ca-certificates curl gnupg \
21,23c27,50
<     && curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh \
<     # Add user "admin" to the Docker group
<     && usermod -a -G docker admin
---
>     && install -m 0755 -d /etc/apt/keyrings \
>     && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
>     && chmod a+r /etc/apt/keyrings/docker.gpg \
>     && echo "deb [arch=$(dpkg --print-architecture) \
>              signed-by=/etc/apt/keyrings/docker.gpg] \
>              https://download.docker.com/linux/ubuntu \
>            $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
>              tee /etc/apt/sources.list.d/docker.list > /dev/null \
>     && apt-get update && apt-cache madison docker-ce \
>     && apt-get install -y docker-ce=5:23.0.6-1~ubuntu.22.04~jammy \
>                           docker-ce-cli=5:23.0.6-1~ubuntu.22.04~jammy \
>                           containerd.io docker-buildx-plugin docker-compose-plugin \
>                           bind9-dnsutils net-tools vim passwd cron rsyslog
>
> # convenience configs
>
> COPY bashprofile /root/.bash_profile
>
> RUN echo 'source ~/.bash_profile' >> /root/.profile \
>     && echo 'colorscheme darkblue' >> /root/.vimrc \
>     && echo '/var/log/* ' > /tmp/t1; cat /etc/logrotate.d/rsyslog >> /tmp/t1; mv -f /tmp/t1 /etc/logrotate.d/rsyslog
>
>
>
28,30c55
<     && rm -rf /var/lib/apt/lists/* \
<     && mkdir /home/admin/.ssh \
<     && chown admin:admin /home/admin/.ssh
---
>     && rm -rf /var/lib/apt/lists/*
31a57
> # SSHd Port
33a60,66
> # Ports for DIND containers  running with --network=host
> EXPOSE 5000-5100
>
> # gitlab registry certificates install
> ADD ./ca.crt /etc/docker/certs.d/myregistry.mydom.mydomain.br:5050/
> ADD ./ca.crt /etc/docker/certs.d/myregistry.mydomain.br:5050/
>
36d68
<

The problem showed up from there. If you understand what happened, could you please explain it to me ?

Here are the tests:

Vanilla test:

wget https://raw.githubusercontent.com/nestybox/dockerfiles/master/ubuntu-jammy-systemd-docker/Dockerfile

mv Dockerfile Dockerfile.vanilla

docker build --tag ubuntu-jammy-systemd-docker:vanilla  . -f ./Dockerfile.vanilla

docker run -it --runtime=sysbox-runc --hostname hmapp03-vanilla --name hmapp03-vanilla --restart=unless-stopped ubuntu-jammy-systemd-docker:vanilla

Welcome to Ubuntu 22.04.3 LTS!

[  ..  ] 

Ubuntu 22.04.3 LTS hmapp03-vanilla console

hmapp03-vanilla login: admin
Password:

Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.5.0-41-generic x86_64)

admin@hmapp03-vanilla:~$ sudo su -
[sudo] password for admin:
root@hmapp03-vanilla:~# ls -la /etc
total 508
drwxr-xr-x 1 root root    4096 Jul 12 17:17 .
drwxr-xr-x 1 root root    4096 Jul 12 17:17 ..
-rw------- 1 root root       0 Jan 25 14:03 .pwd.lock
drwxr-xr-x 3 root root    4096 Jul 12 17:17 X11
-rw-r--r-- 1 root root    3028 Jan 25 14:03 adduser.conf
drwxr-xr-x 1 root root    4096 Jul 12 17:17 alternatives
drwxr-xr-x 2 root root    4096 Jul 12 17:17 apparmor
drwxr-xr-x 8 root root    4096 Jul 12 17:17 apparmor.d
drwxr-xr-x 1 root root    4096 Jan 25 14:03 apt
-rw-r--r-- 1 root root    2319 Jan  6  2022 bash.bashrc
drwxr-xr-x 1 root root    4096 Jul 12 17:17 bash_completion.d
-rw-r--r-- 1 root root     367 Dec 16  2020 bindresvport.blacklist
drwxr-xr-x 2 root root    4096 Nov 21  2023 binfmt.d
drwxr-xr-x 3 root root    4096 Mar  3 07:00 ca-certificates
-rw-r--r-- 1 root root    5892 Mar  3 07:00 ca-certificates.conf
drwxr-xr-x 2 root root    4096 Jan 25 14:06 cloud
drwxr-xr-x 2 root root    4096 Jul 12 17:17 containerd
drwxr-xr-x 2 root root    4096 Jan 25 14:06 cron.d
drwxr-xr-x 2 root root    4096 Jan 25 14:06 cron.daily
drwxr-xr-x 4 root root    4096 Mar  3 07:00 dbus-1
-rw-r--r-- 1 root root    2969 Feb 20  2022 debconf.conf
-rw-r--r-- 1 root root      13 Aug 22  2021 debian_version
drwxr-xr-x 1 root root    4096 Jul 12 17:17 default
-rw-r--r-- 1 root root     604 Sep 15  2018 deluser.conf
drwxr-xr-x 2 root root    4096 Mar  3 07:00 depmod.d
drwxr-xr-x 2 root root    4096 Jun 29 00:02 docker
drwxr-xr-x 4 root root    4096 Jan 25 14:06 dpkg
-rw-r--r-- 1 root root     685 Jan  8  2022 e2scrub.conf
-rw-r--r-- 1 root root     106 Jan 25 14:03 environment
-rw-r--r-- 1 root root    1816 Dec 27  2019 ethertypes
-rw-r--r-- 1 root root      37 Jan 25 14:03 fstab
-rw-r--r-- 1 root root    2584 Feb  3  2022 gai.conf
-rw-r--r-- 1 root root     632 Jul 12 17:17 group
-rw-r--r-- 1 root root     627 Jul 12 17:17 group-
-rw-r----- 1 root shadow   526 Jul 12 17:17 gshadow
-rw-r----- 1 root shadow   521 Jul 12 17:17 gshadow-
drwxr-xr-x 3 root root    4096 Feb 21  2022 gss
-rw-r--r-- 1 root root      92 Oct 15  2021 host.conf
-rw-r--r-- 1 root root      16 Jul 12 17:17 hostname
-rw-r--r-- 1 root root     179 Jul 12 17:17 hosts
-rw-r--r-- 1 root root     411 Jul 12 17:17 hosts.allow
-rw-r--r-- 1 root root     711 Jul 12 17:17 hosts.deny
drwxr-xr-x 1 root root    4096 Jul 12 17:17 init.d
drwxr-xr-x 4 root root    4096 Mar  3 07:00 iproute2
-rw-r--r-- 1 root root      26 Jan  2  2024 issue
-rw-r--r-- 1 root root      19 Jan  2  2024 issue.net
drwxr-xr-x 1 root root    4096 Mar  3 07:00 kernel
-rw-r--r-- 1 root root    9191 Jul 12 17:17 ld.so.cache
-rw-r--r-- 1 root root      34 Dec 16  2020 ld.so.conf
drwxr-xr-x 2 root root    4096 Jan 25 14:06 ld.so.conf.d
drwxr-xr-x 2 root root    4096 Jul 12 17:16 ldap
-rw-r--r-- 1 root root     267 Oct 15  2021 legal
-rw-r--r-- 1 root root     191 Mar 17  2022 libaudit.conf
-rw-r--r-- 1 root root    2996 Jan  2  2024 locale.alias
-rw-r--r-- 1 root root    9458 Mar  3 07:00 locale.gen
drwxr-xr-x 3 root root    4096 Jul 12 17:16 logcheck
-rw-r--r-- 1 root root   10734 Nov 11  2021 login.defs
drwxr-xr-x 2 root root    4096 Jan 25 14:06 logrotate.d
-rw-r--r-- 1 root root     104 Jan  2  2024 lsb-release
-rw-r--r-- 1 root root      33 Mar  3 07:00 machine-id
-rw-r--r-- 1 root root     744 Jan  8  2022 mke2fs.conf
drwxr-xr-x 2 root root    4096 Mar  3 07:00 modprobe.d
-rw-r--r-- 1 root root     195 Mar  3 07:00 modules
drwxr-xr-x 2 root root    4096 Mar  3 07:00 modules-load.d
lrwxrwxrwx 1 root root      19 Jul 12 17:17 mtab -> ../proc/self/mounts
-rw-r--r-- 1 root root     767 Mar 24  2022 netconfig
-rw-r--r-- 1 root root      91 Oct 15  2021 networks
-rw-r--r-- 1 root root     494 Dec 16  2020 nsswitch.conf
drwxr-xr-x 2 root root    4096 Jan 25 14:03 opt
lrwxrwxrwx 1 root root      21 Jan  2  2024 os-release -> ../usr/lib/os-release
-rw-r--r-- 1 root root     552 Aug 12  2020 pam.conf
drwxr-xr-x 1 root root    4096 Jul 12 17:17 pam.d
-rw-r--r-- 1 root root    1226 Jul 12 17:17 passwd
-rw-r--r-- 1 root root    1226 Jul 12 17:17 passwd-
drwxr-xr-x 3 root root    4096 Jul 12 17:17 perl
-rw-r--r-- 1 root root     582 Oct 15  2021 profile
drwxr-xr-x 2 root root    4096 Jan 25 14:06 profile.d
-rw-r--r-- 1 root root    2932 Apr  1  2013 protocols
drwxr-xr-x 1 root root    4096 Jul 12 17:17 rc0.d
drwxr-xr-x 1 root root    4096 Jul 12 17:17 rc1.d
drwxr-xr-x 1 root root    4096 Jul 12 17:17 rc2.d
drwxr-xr-x 1 root root    4096 Jul 12 17:17 rc3.d
drwxr-xr-x 1 root root    4096 Jul 12 17:17 rc4.d
drwxr-xr-x 1 root root    4096 Jul 12 17:17 rc5.d
drwxr-xr-x 1 root root    4096 Jul 12 17:17 rc6.d
drwxr-xr-x 1 root root    4096 Jul 12 17:17 rcS.d
-rw-r--r-- 1 root root      26 Jul 12 17:17 resolv.conf
lrwxrwxrwx 1 root root      13 Dec  5  2023 rmt -> /usr/sbin/rmt
-rw-r--r-- 1 root root     887 Apr  1  2013 rpc
drwxr-xr-x 4 root root    4096 Jan 25 14:06 security
drwxr-xr-x 2 root root    4096 Jan 25 14:05 selinux
-rw-r--r-- 1 root root   12813 Mar 27  2021 services
-rw-r----- 1 root shadow   732 Jul 12 17:17 shadow
-rw-r----- 1 root shadow   732 Jul 12 17:17 shadow-
-rw-r--r-- 1 root root     128 Jan 25 14:03 shells
drwxr-xr-x 2 root root    4096 Jan 25 14:03 skel
drwxr-xr-x 1 root root    4096 Jul 12 17:17 ssh
drwxr-xr-x 4 root root    4096 Mar  3 07:00 ssl
-rw-r--r-- 1 root root      19 Mar  3 07:00 subgid
-rw-r--r-- 1 root root       0 Jan 25 14:03 subgid-
-rw-r--r-- 1 root root      19 Mar  3 07:00 subuid
-rw-r--r-- 1 root root       0 Jan 25 14:03 subuid-
-rw-r--r-- 1 root root    4573 Apr  3  2023 sudo.conf
-rw-r--r-- 1 root root    9390 Apr  3  2023 sudo_logsrvd.conf
-r--r----- 1 root root    1671 Aug  3  2022 sudoers
drwxr-xr-x 2 root root    4096 Mar  3 07:00 sudoers.d
-rw-r--r-- 1 root root    2355 Feb 25  2022 sysctl.conf
drwxr-xr-x 1 root root    4096 Mar  3 07:00 sysctl.d
drwxr-xr-x 1 root root    4096 Mar  3 07:00 systemd
drwxr-xr-x 2 root root    4096 Jan 25 14:06 terminfo
drwxr-xr-x 2 root root    4096 Nov 21  2023 tmpfiles.d
-rw-r--r-- 1 root root    1260 Jun 16  2020 ucf.conf
drwxr-xr-x 4 root root    4096 Mar  3 07:00 udev
drwxr-xr-x 3 root root    4096 Jul 12 17:17 ufw
drwxr-xr-x 2 root root    4096 Jan 25 14:06 update-motd.d
-rw-r--r-- 1 root root     681 Mar 23  2022 xattr.conf
drwxr-xr-x 1 root root    4096 Jul 12 17:17 xdg

root@hmapp03-vanilla:~#

V4 custom image test:

# docker run -it --runtime=sysbox-runc --hostname hmapp03-v4 --name hmapp03-v4 --restart=unless-stopped ubuntu-jammy-systemd-docker:v4

Welcome to Ubuntu 22.04.3 LTS!

[  ..  ] 

Ubuntu 22.04.3 LTS hmapp03-v4 console

hmapp03-v4 login: root
Password:
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 6.5.0-41-generic x86_64)

________
13:40:07 root@hmapp03-v4:~
# ls -la /etc
total 620
drwxr-xr-x 1 root   root     4096 Jul 12 13:39 .
drwxr-xr-x 1 root   root     4096 Jul 12 13:40 ..
-rw------- 1 root   root        0 Oct  4  2023 .pwd.lock
drwxr-xr-x 3 nobody nogroup  4096 Jan  7  2024 X11
-rw-r--r-- 1 root   root     3028 Oct  4  2023 adduser.conf
drwxr-xr-x 1 root   root     4096 Jan  7  2024 alternatives
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 apparmor
drwxr-xr-x 8 nobody nogroup  4096 Jan  7  2024 apparmor.d
drwxr-xr-x 1 root   root     4096 Oct  4  2023 apt
-rw-r--r-- 1 root   root     2319 Jan  6  2022 bash.bashrc
drwxr-xr-x 1 root   root     4096 Jan  7  2024 bash_completion.d
-rw-r--r-- 1 root   root      367 Dec 16  2020 bindresvport.blacklist
drwxr-xr-x 2 root   root     4096 Sep 19  2023 binfmt.d
drwxr-xr-x 3 root   root     4096 Nov 30  2023 ca-certificates
-rw-r--r-- 1 root   root     5892 Nov 30  2023 ca-certificates.conf
drwxr-xr-x 2 nobody nogroup  4096 Oct  4  2023 cloud
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 containerd
drwxr-xr-x 1 root   root     4096 Jan  7  2024 cron.d
drwxr-xr-x 1 root   root     4096 Jan  7  2024 cron.daily
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 cron.hourly
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 cron.monthly
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 cron.weekly
-rw-r--r-- 1 root   root     1136 Mar 23  2022 crontab
drwxr-xr-x 4 root   root     4096 Nov 30  2023 dbus-1
-rw-r--r-- 1 root   root     2969 Feb 20  2022 debconf.conf
-rw-r--r-- 1 root   root       13 Aug 22  2021 debian_version
drwxr-xr-x 1 root   root     4096 Jan  7  2024 default
-rw-r--r-- 1 root   root      604 Sep 15  2018 deluser.conf
drwxr-xr-x 2 nobody nogroup  4096 Nov 30  2023 depmod.d
drwxr-xr-x 1 root   root     4096 Jan  7  2024 docker
drwxr-xr-x 4 nobody nogroup  4096 Oct  4  2023 dpkg
-rw-r--r-- 1 root   root      685 Jan  8  2022 e2scrub.conf
-rw-r--r-- 1 root   root      106 Oct  4  2023 environment
-rw-r--r-- 1 root   root     1816 Dec 27  2019 ethertypes
-rw-r--r-- 1 root   root       37 Oct  4  2023 fstab
-rw-r--r-- 1 root   root     2584 Feb  3  2022 gai.conf
-rw-r--r-- 1 root   root      643 Jan  7  2024 group
-rw-r--r-- 1 root   root      629 Jan  7  2024 group-
-rw-r----- 1 root   shadow    535 Jan  7  2024 gshadow
-rw-r----- 1 root   shadow    524 Jan  7  2024 gshadow-
drwxr-xr-x 3 root   root     4096 Feb 21  2022 gss
-rw-r--r-- 1 root   root       92 Oct 15  2021 host.conf
-rw-r--r-- 1 root   root       11 Jul 12 13:39 hostname
-rw-r--r-- 1 root   root      174 Jul 12 13:39 hosts
-rw-r--r-- 1 root   root      411 Jan  7  2024 hosts.allow
-rw-r--r-- 1 root   root      711 Jan  7  2024 hosts.deny
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 init
drwxr-xr-x 1 root   root     4096 Jan  7  2024 init.d
-rw-r--r-- 1 root   root     1748 Jan  6  2022 inputrc
drwxr-xr-x 4 nobody nogroup  4096 Nov 30  2023 iproute2
-rw-r--r-- 1 root   root       26 Aug  2  2023 issue
-rw-r--r-- 1 root   root       19 Aug  2  2023 issue.net
drwxr-xr-x 1 root   root     4096 Nov 30  2023 kernel
-rw-r--r-- 1 root   root    10847 Jan  7  2024 ld.so.cache
-rw-r--r-- 1 root   root       34 Dec 16  2020 ld.so.conf
drwxr-xr-x 2 nobody nogroup  4096 Oct  4  2023 ld.so.conf.d
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 ldap
-rw-r--r-- 1 root   root      267 Oct 15  2021 legal
-rw-r--r-- 1 root   root      191 Mar 17  2022 libaudit.conf
-rw-r--r-- 1 root   root     2996 Sep 25  2023 locale.alias
-rw-r--r-- 1 root   root     9458 Nov 30  2023 locale.gen
drwxr-xr-x 3 nobody nogroup  4096 Jan  7  2024 logcheck
-rw-r--r-- 1 root   root    10734 Nov 11  2021 login.defs
-rw-r--r-- 1 root   root      592 May 25  2022 logrotate.conf
drwxr-xr-x 1 root   root     4096 Jan  7  2024 logrotate.d
-rw-r--r-- 1 root   root      104 Aug  2  2023 lsb-release
-rw-r--r-- 1 root   root       33 Nov 30  2023 machine-id
-rw-r--r-- 1 root   root    72029 Mar 21  2022 mime.types
-rw-r--r-- 1 root   root      744 Jan  8  2022 mke2fs.conf
drwxr-xr-x 2 nobody nogroup  4096 Nov 30  2023 modprobe.d
-rw-r--r-- 1 root   root      195 Nov 30  2023 modules
drwxr-xr-x 2 nobody nogroup  4096 Nov 30  2023 modules-load.d
lrwxrwxrwx 1 root   root       19 Jul 12 13:39 mtab -> ../proc/self/mounts
-rw-r--r-- 1 root   root      767 Mar 24  2022 netconfig
-rw-r--r-- 1 root   root       91 Oct 15  2021 networks
-rw-r--r-- 1 root   root      494 Dec 16  2020 nsswitch.conf
drwxr-xr-x 2 root   root     4096 Oct  4  2023 opt
lrwxrwxrwx 1 root   root       21 Aug  2  2023 os-release -> ../usr/lib/os-release
-rw-r--r-- 1 root   root      552 Aug 12  2020 pam.conf
drwxr-xr-x 1 root   root     4096 Jan  7  2024 pam.d
-rw-r--r-- 1 root   root     1234 Jan  7  2024 passwd
-rw-r--r-- 1 root   root     1234 Jan  7  2024 passwd-
drwxr-xr-x 3 nobody nogroup  4096 Jan  7  2024 perl
-rw-r--r-- 1 root   root      582 Oct 15  2021 profile
drwxr-xr-x 2 nobody nogroup  4096 Oct  4  2023 profile.d
-rw-r--r-- 1 root   root     2932 Apr  1  2013 protocols
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 python3.10
drwxr-xr-x 1 root   root     4096 Jan  7  2024 rc0.d
drwxr-xr-x 1 root   root     4096 Jan  7  2024 rc1.d
drwxr-xr-x 1 root   root     4096 Jan  7  2024 rc2.d
drwxr-xr-x 1 root   root     4096 Jan  7  2024 rc3.d
drwxr-xr-x 1 root   root     4096 Jan  7  2024 rc4.d
drwxr-xr-x 1 root   root     4096 Jan  7  2024 rc5.d
drwxr-xr-x 1 root   root     4096 Jan  7  2024 rc6.d
drwxr-xr-x 1 root   root     4096 Jan  7  2024 rcS.d
-rw-r--r-- 1 root   root       26 Jul 12 13:39 resolv.conf
lrwxrwxrwx 1 root   root       13 Feb 15  2023 rmt -> /usr/sbin/rmt
-rw-r--r-- 1 root   root      887 Apr  1  2013 rpc
-rw-r--r-- 1 root   root     1382 Dec 23  2021 rsyslog.conf
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 rsyslog.d
drwxr-xr-x 4 nobody nogroup  4096 Oct  4  2023 security
drwxr-xr-x 2 nobody nogroup  4096 Oct  4  2023 selinux
-rw-r--r-- 1 root   root    12813 Mar 27  2021 services
-rw-r----- 1 root   shadow    733 Jan  7  2024 shadow
-rw-r----- 1 root   shadow    733 Jan  7  2024 shadow-
-rw-r--r-- 1 root   root      128 Oct  4  2023 shells
drwxr-xr-x 2 nobody nogroup  4096 Oct  4  2023 skel
drwxr-xr-x 1 root   root     4096 Jan  7  2024 ssh
drwxr-xr-x 4 nobody nogroup  4096 Nov 30  2023 ssl
-rw-r--r-- 1 root   root        0 Oct  4  2023 subgid
-rw-r--r-- 1 root   root        0 Oct  4  2023 subuid
-rw-r--r-- 1 root   root     4573 Apr  3  2023 sudo.conf
-rw-r--r-- 1 root   root     9390 Apr  3  2023 sudo_logsrvd.conf
-r--r----- 1 root   root     1671 Aug  3  2022 sudoers
drwxr-xr-x 2 nobody nogroup  4096 Nov 30  2023 sudoers.d
-rw-r--r-- 1 root   root     2355 Feb 25  2022 sysctl.conf
drwxr-xr-x 1 root   root     4096 Nov 30  2023 sysctl.d
drwxr-xr-x 1 root   root     4096 Nov 30  2023 systemd
drwxr-xr-x 2 nobody nogroup  4096 Oct  4  2023 terminfo
drwxr-xr-x 2 root   root     4096 Sep 19  2023 tmpfiles.d
-rw-r--r-- 1 root   root     1260 Jun 16  2020 ucf.conf
drwxr-xr-x 4 nobody nogroup  4096 Nov 30  2023 udev
drwxr-xr-x 3 root   root     4096 Jan  7  2024 ufw
drwxr-xr-x 2 nobody nogroup  4096 Oct  4  2023 update-motd.d
drwxr-xr-x 2 nobody nogroup  4096 Jan  7  2024 vim
-rw-r--r-- 1 root   root      681 Mar 23  2022 xattr.conf
drwxr-xr-x 1 root   root     4096 Jan  7  2024 xdg
________
13:40:10 root@hmapp03-v4:~
#
ffabreti commented 1 month ago

I have noticed that all packages installed from image v4 ended up creating something in /etc that has added or changed the date of directories AND set owner to nobody: image

I've been thinking if creating a "admin" user has any part on the permission problem (since I removed that user)

ffabreti commented 1 month ago

New test with admin user restored

#   diff --color Dockerfile.vanilla Dockerfile.admin
18,19c18,19
< # Install Docker
< RUN apt-get update && apt-get install -y curl \
---
> # Install Docker and utils install
> RUN apt-get update && apt-get install -y ca-certificates curl gnupg \
21c21,33
<     && curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh \
---
>     && install -m 0755 -d /etc/apt/keyrings \
>     && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
>     && chmod a+r /etc/apt/keyrings/docker.gpg \
>     && echo "deb [arch=$(dpkg --print-architecture) \
>              signed-by=/etc/apt/keyrings/docker.gpg] \
>              https://download.docker.com/linux/ubuntu \
>            $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
>              tee /etc/apt/sources.list.d/docker.list > /dev/null \
>     && apt-get update && apt-cache madison docker-ce \
>     && apt-get install -y docker-ce=5:23.0.6-1~ubuntu.22.04~jammy \
>                           docker-ce-cli=5:23.0.6-1~ubuntu.22.04~jammy \
>                           containerd.io docker-buildx-plugin docker-compose-plugin \
>                           bind9-dnsutils net-tools vim passwd cron rsyslog \
23a36,45
>
> # configs de conveniencia
>
> COPY bashprofile /root/.bash_profile
>
> RUN echo 'source ~/.bash_profile' >> /home/admin/.profile \
>     && echo 'colorscheme darkblue' >> /home/admin/.vimrc \
>     && echo '/var/log/* ' > /tmp/t1; cat /etc/logrotate.d/rsyslog >> /tmp/t1; mv -f /tmp/t1 /etc/logrotate.d/rsyslog
>
>
31a54
> # Porta SSHd
33a57,63
> # Portas para conteiners DIND rodando com --network=host
> EXPOSE 5000-5100
>
> # gitlab registry certificates install
> ADD ./ca.crt /etc/docker/certs.d/registry.gitlab.hm.sit.br:5050/
> ADD ./ca.crt /etc/docker/certs.d/registry.gitlab.sit.br:5050/
>
36d65
<
# docker build --tag ubuntu-jammy-systemd-docker:admin . -f ./Dockerfile.admin

# docker run -it --runtime=sysbox-runc --hostname hmapp03-admin --name hmapp03-admin ubuntu-jammy-systemd-docker:admin

Welcome to Ubuntu 22.04.3 LTS!

[  ..  ] 

Ubuntu 22.04.3 LTS hmapp03-admin console

hmapp03-admin login: admin
Password:

-bash: /home/admin/.bash_profile: No such file or directory
admin@hmapp03-admin:~$ sudo su -
[sudo] password for admin:
________
18:47:15 root@hmapp03-admin:~
# ls -la /etc
total 628
drwxr-xr-x 1 root root    4096 Jul 16 18:46 .
drwxr-xr-x 1 root root    4096 Jul 16 18:46 ..
-rw------- 1 root root       0 Jan 25 14:03 .pwd.lock
drwxr-xr-x 3 root root    4096 Jul 16 18:44 X11
-rw-r--r-- 1 root root    3028 Jan 25 14:03 adduser.conf
drwxr-xr-x 1 root root    4096 Jul 16 18:44 alternatives
drwxr-xr-x 2 root root    4096 Jul 16 18:44 apparmor
drwxr-xr-x 8 root root    4096 Jul 16 18:44 apparmor.d
drwxr-xr-x 1 root root    4096 Jan 25 14:03 apt
-rw-r--r-- 1 root root    2319 Jan  6  2022 bash.bashrc
drwxr-xr-x 1 root root    4096 Jul 16 18:44 bash_completion.d
-rw-r--r-- 1 root root     367 Dec 16  2020 bindresvport.blacklist
drwxr-xr-x 2 root root    4096 Nov 21  2023 binfmt.d
drwxr-xr-x 3 root root    4096 Mar  3 07:00 ca-certificates
-rw-r--r-- 1 root root    5892 Mar  3 07:00 ca-certificates.conf
drwxr-xr-x 2 root root    4096 Jan 25 14:06 cloud
drwxr-xr-x 2 root root    4096 Jul 16 18:44 containerd
drwxr-xr-x 1 root root    4096 Jul 16 18:44 cron.d
drwxr-xr-x 1 root root    4096 Jul 16 18:44 cron.daily
drwxr-xr-x 2 root root    4096 Jul 16 18:44 cron.hourly
drwxr-xr-x 2 root root    4096 Jul 16 18:44 cron.monthly
drwxr-xr-x 2 root root    4096 Jul 16 18:44 cron.weekly
-rw-r--r-- 1 root root    1136 Mar 23  2022 crontab
drwxr-xr-x 4 root root    4096 Mar  3 07:00 dbus-1
-rw-r--r-- 1 root root    2969 Feb 20  2022 debconf.conf
-rw-r--r-- 1 root root      13 Aug 22  2021 debian_version
drwxr-xr-x 1 root root    4096 Jul 16 18:44 default
-rw-r--r-- 1 root root     604 Sep 15  2018 deluser.conf
drwxr-xr-x 2 root root    4096 Mar  3 07:00 depmod.d
drwxr-xr-x 1 root root    4096 Jul 16 18:44 docker
drwxr-xr-x 4 root root    4096 Jan 25 14:06 dpkg
-rw-r--r-- 1 root root     685 Jan  8  2022 e2scrub.conf
-rw-r--r-- 1 root root     106 Jan 25 14:03 environment
-rw-r--r-- 1 root root    1816 Dec 27  2019 ethertypes
-rw-r--r-- 1 root root      37 Jan 25 14:03 fstab
-rw-r--r-- 1 root root    2584 Feb  3  2022 gai.conf
-rw-r--r-- 1 root root     667 Jul 16 18:44 group
-rw-r--r-- 1 root root     662 Jul 16 18:44 group-
-rw-r----- 1 root shadow   555 Jul 16 18:44 gshadow
-rw-r----- 1 root shadow   550 Jul 16 18:44 gshadow-
drwxr-xr-x 3 root root    4096 Feb 21  2022 gss
-rw-r--r-- 1 root root      92 Oct 15  2021 host.conf
-rw-r--r-- 1 root root      14 Jul 16 18:46 hostname
-rw-r--r-- 1 root root     177 Jul 16 18:46 hosts
-rw-r--r-- 1 root root     411 Jul 16 18:44 hosts.allow
-rw-r--r-- 1 root root     711 Jul 16 18:44 hosts.deny
drwxr-xr-x 2 root root    4096 Jul 16 18:44 init
drwxr-xr-x 1 root root    4096 Jul 16 18:44 init.d
-rw-r--r-- 1 root root    1748 Jan  6  2022 inputrc
drwxr-xr-x 4 root root    4096 Mar  3 07:00 iproute2
-rw-r--r-- 1 root root      26 Jan  2  2024 issue
-rw-r--r-- 1 root root      19 Jan  2  2024 issue.net
drwxr-xr-x 1 root root    4096 Mar  3 07:00 kernel
-rw-r--r-- 1 root root   10847 Jul 16 18:44 ld.so.cache
-rw-r--r-- 1 root root      34 Dec 16  2020 ld.so.conf
drwxr-xr-x 2 root root    4096 Jan 25 14:06 ld.so.conf.d
drwxr-xr-x 2 root root    4096 Jul 16 18:44 ldap
-rw-r--r-- 1 root root     267 Oct 15  2021 legal
-rw-r--r-- 1 root root     191 Mar 17  2022 libaudit.conf
-rw-r--r-- 1 root root    2996 Jan  2  2024 locale.alias
-rw-r--r-- 1 root root    9458 Mar  3 07:00 locale.gen
drwxr-xr-x 3 root root    4096 Jul 16 18:44 logcheck
-rw-r--r-- 1 root root   10734 Nov 11  2021 login.defs
-rw-r--r-- 1 root root     592 May 25  2022 logrotate.conf
drwxr-xr-x 1 root root    4096 Jul 16 18:44 logrotate.d
-rw-r--r-- 1 root root     104 Jan  2  2024 lsb-release
-rw-r--r-- 1 root root      33 Mar  3 07:00 machine-id
-rw-r--r-- 1 root root   72029 Mar 21  2022 mime.types
-rw-r--r-- 1 root root     744 Jan  8  2022 mke2fs.conf
drwxr-xr-x 2 root root    4096 Mar  3 07:00 modprobe.d
-rw-r--r-- 1 root root     195 Mar  3 07:00 modules
drwxr-xr-x 2 root root    4096 Mar  3 07:00 modules-load.d
lrwxrwxrwx 1 root root      19 Jul 16 18:46 mtab -> ../proc/self/mounts
-rw-r--r-- 1 root root     767 Mar 24  2022 netconfig
-rw-r--r-- 1 root root      91 Oct 15  2021 networks
-rw-r--r-- 1 root root     494 Dec 16  2020 nsswitch.conf
drwxr-xr-x 2 root root    4096 Jan 25 14:03 opt
lrwxrwxrwx 1 root root      21 Jan  2  2024 os-release -> ../usr/lib/os-release
-rw-r--r-- 1 root root     552 Aug 12  2020 pam.conf
drwxr-xr-x 1 root root    4096 Jul 16 18:44 pam.d
-rw-r--r-- 1 root root    1275 Jul 16 18:44 passwd
-rw-r--r-- 1 root root    1275 Jul 16 18:44 passwd-
drwxr-xr-x 3 root root    4096 Jul 16 18:44 perl
-rw-r--r-- 1 root root     582 Oct 15  2021 profile
drwxr-xr-x 2 root root    4096 Jan 25 14:06 profile.d
-rw-r--r-- 1 root root    2932 Apr  1  2013 protocols
drwxr-xr-x 2 root root    4096 Jul 16 18:44 python3.10
drwxr-xr-x 1 root root    4096 Jul 16 18:44 rc0.d
drwxr-xr-x 1 root root    4096 Jul 16 18:44 rc1.d
drwxr-xr-x 1 root root    4096 Jul 16 18:44 rc2.d
drwxr-xr-x 1 root root    4096 Jul 16 18:44 rc3.d
drwxr-xr-x 1 root root    4096 Jul 16 18:44 rc4.d
drwxr-xr-x 1 root root    4096 Jul 16 18:44 rc5.d
drwxr-xr-x 1 root root    4096 Jul 16 18:44 rc6.d
drwxr-xr-x 1 root root    4096 Jul 16 18:44 rcS.d
-rw-r--r-- 1 root root      26 Jul 16 18:46 resolv.conf
lrwxrwxrwx 1 root root      13 Dec  5  2023 rmt -> /usr/sbin/rmt
-rw-r--r-- 1 root root     887 Apr  1  2013 rpc
-rw-r--r-- 1 root root    1382 Dec 23  2021 rsyslog.conf
drwxr-xr-x 2 root root    4096 Jul 16 18:44 rsyslog.d
drwxr-xr-x 4 root root    4096 Jan 25 14:06 security
drwxr-xr-x 2 root root    4096 Jan 25 14:05 selinux
-rw-r--r-- 1 root root   12813 Mar 27  2021 services
-rw-r----- 1 root shadow   760 Jul 16 18:44 shadow
-rw-r----- 1 root shadow   760 Jul 16 18:44 shadow-
-rw-r--r-- 1 root root     128 Jan 25 14:03 shells
drwxr-xr-x 2 root root    4096 Jan 25 14:03 skel
drwxr-xr-x 1 root root    4096 Jul 16 18:44 ssh
drwxr-xr-x 4 root root    4096 Mar  3 07:00 ssl
-rw-r--r-- 1 root root      19 Mar  3 07:00 subgid
-rw-r--r-- 1 root root       0 Jan 25 14:03 subgid-
-rw-r--r-- 1 root root      19 Mar  3 07:00 subuid
-rw-r--r-- 1 root root       0 Jan 25 14:03 subuid-
-rw-r--r-- 1 root root    4573 Apr  3  2023 sudo.conf
-rw-r--r-- 1 root root    9390 Apr  3  2023 sudo_logsrvd.conf
-r--r----- 1 root root    1671 Aug  3  2022 sudoers
drwxr-xr-x 2 root root    4096 Mar  3 07:00 sudoers.d
-rw-r--r-- 1 root root    2355 Feb 25  2022 sysctl.conf
drwxr-xr-x 1 root root    4096 Mar  3 07:00 sysctl.d
drwxr-xr-x 1 root root    4096 Mar  3 07:00 systemd
drwxr-xr-x 2 root root    4096 Jan 25 14:06 terminfo
drwxr-xr-x 2 root root    4096 Nov 21  2023 tmpfiles.d
-rw-r--r-- 1 root root    1260 Jun 16  2020 ucf.conf
drwxr-xr-x 4 root root    4096 Mar  3 07:00 udev
drwxr-xr-x 3 root root    4096 Jul 16 18:44 ufw
drwxr-xr-x 2 root root    4096 Jan 25 14:06 update-motd.d
drwxr-xr-x 2 root root    4096 Jul 16 18:44 vim
-rw-r--r-- 1 root root     681 Mar 23  2022 xattr.conf
drwxr-xr-x 1 root root    4096 Jul 16 18:44 xdg
ctalledo commented 1 month ago

Thanks @ffabreti for the latest info.

I've been thinking if creating a "admin" user has any part on the permission problem (since I removed that user)

So if I understand correctly, that seems to have been the problem correct (given that your last test above looks much better).

Let me know please.

Thanks!

ffabreti commented 1 month ago

I have done a script to workaround the issue while I cannot reinstall the container. Not exactly sure of what I'm doing here, so (googlers) be warned.

https://gist.github.com/ffabreti/c9ad7b882118fa0106ccbfbf3942bcfd