search

rootless-containers / usernetes

Kubernetes without the root privileges
https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2033-kubelet-in-userns-aka-rootless
Apache License 2.0
865 stars 58 forks source link

Deriving my own Docker image #239

Closed dg424 closed 3 years ago

dg424 commented 3 years ago

Hi,

I'm trying to build my own image using the main usernetes image as the base image but it doesn't start. The Dockerfile has one line:

FROM ghcr.io/rootless-containers/usernetes:master

Running with the same sample line from the docs:

docker run -td --name usernetes-node -p 127.0.0.1:6443:6443 --privileged usernetes --cri=containerd

Gives the following errors;

Aug 25 14:43:49 1dded8a4ab08 systemd[1]: Started Session c1 of User user. Aug 25 14:43:49 1dded8a4ab08 systemd[50]: pam_unix(login:session): session opened for user user(uid=1000) by (uid=0) exit 1 Connection to the local host terminated. Sending SIGTERM to remaining processes... Sending SIGKILL to remaining processes... All filesystems, swaps, loop devices, MD devices and DM devices detached. Exiting container.

Any ideas ?

AkihiroSuda commented 3 years ago

Could you provide docker info and docker version

dg424 commented 3 years ago

$ docker info Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Build with BuildKit (Docker Inc., v0.6.1-docker) compose: Docker Compose (Docker Inc., v2.0.0-beta.6) scan: Docker Scan (Docker Inc., v0.8.0)

Server: Containers: 19 Running: 11 Paused: 0 Stopped: 8 Images: 273 Server Version: 20.10.7 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 5.10.43.3-microsoft-standard-WSL2 Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.63GiB Name: docker-desktop ID: MBSU:OYE3:SYPP:D2E5:4BWY:XEHE:KV4O:CVUP:UYNQ:34BS:JRYO:JTDR Docker Root Dir: /var/lib/docker Debug Mode: true File Descriptors: 112673 Goroutines: 112665 System Time: 2021-08-26T02:45:06.5133409Z EventsListeners: 3 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support WARNING: No blkio throttle.write_bps_device support WARNING: No blkio throttle.read_iops_device support WARNING: No blkio throttle.write_iops_device support

$ docker version Client: Docker Engine - Community Version: 20.10.8 API version: 1.41 Go version: go1.16.6 Git commit: 3967b7d Built: Fri Jul 30 19:54:27 2021 OS/Arch: linux/amd64 Context: default Experimental: true

Server: Docker Engine - Community Engine: Version: 20.10.7 API version: 1.41 (minimum version 1.12) Go version: go1.13.15 Git commit: b0f5bc3 Built: Wed Jun 2 11:54:58 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.6 GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d runc: Version: 1.0.0-rc95 GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 docker-init: Version: 0.19.0 GitCommit: de40ad0

Server is docker desktop.

AkihiroSuda commented 3 years ago

Cgroup Version: 1

Usernetes v20210708.0 is the last version that supports cgroup v1. The current master requires cgroup v2. (I don't use WSL and I don't know how to configure WSL to use cgroup v2)

dg424 commented 3 years ago

But when I run it directly, not using a derived container, it works:

$ docker run -it --name usernetes-node -p 127.0.0.1:6443:6443 --
privileged ghcr.io/rootless-containers/usernetes --cri=containerd
Created symlink /etc/systemd/system/systemd-firstboot.service → /dev/null.
Created symlink /etc/systemd/system/systemd-udevd.service → /dev/null.
Created symlink /etc/systemd/system/multi-user.target.wants/docker-entrypoint.service → /etc/systemd/system/docker-entrypoint.service.
/docker-entrypoint.sh: starting /lib/systemd/systemd --show-status=false --unit=docker-entrypoint.target
systemd v248.3-1.fc34 running in system mode. (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization wsl.
Detected architecture x86-64.
Failed to create symlink /sys/fs/cgroup/net_cls: File exists
Failed to create symlink /sys/fs/cgroup/net_prio: File exists
Failed to create symlink /sys/fs/cgroup/cpuacct: File exists
Failed to create symlink /sys/fs/cgroup/cpu: File exists
modprobe@configfs.service: Deactivated successfully.
modprobe@drm.service: Deactivated successfully.
modprobe@fuse.service: Deactivated successfully.
+ source /etc/docker-entrypoint-cmd
++ unsudo /home/user/usernetes/boot/docker-2ndboot.sh --cri=containerd
+ car=/home/user/usernetes/boot/docker-2ndboot.sh
+ shift
+ cdr=--cri=containerd
++ which /home/user/usernetes/boot/docker-2ndboot.sh
+ exec machinectl shell user@ /home/user/usernetes/boot/docker-2ndboot.sh --cri=containerd
Connected to the local host. Press ^] three times within 1s to exit session.
+ ./install.sh --cri=containerd
[WARNING] Disabling Rootless cgroup: the system is using cgroup v1, you need to reboot the system with systemd.unified_cgroup_hierarchy=1
[WARNING] Cgroup is disabled. In future version of Usernetes, cgroup (v2) will be an essential requirement.
[INFO] Generating single-node cluster TLS keys (/home/user/.config/usernetes/{master,node})
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/{ca.pem,ca-key.pem}
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/{admin.pem,admin-key.pem}
2021/08/27 12:52:52 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/admin-localhost.kubeconfig
Cluster "kubernetes-the-hard-way" set.
User "admin" set.
Context "default" created.
Switched to context "default".
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/admin-127.0.0.1.kubeconfig
Cluster "kubernetes-the-hard-way" set.
User "admin" set.
Context "default" created.
Switched to context "default".
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/{kube-controller-manager.pem,kube-controller-manager-key.pem}
2021/08/27 12:52:53 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/kube-controller-manager.kubeconfig
Cluster "kubernetes-the-hard-way" set.
User "system:kube-controller-manager" set.
Context "default" created.
Switched to context "default".
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/{kube-proxy.pem,kube-proxy-key.pem}
2021/08/27 12:52:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/kube-proxy.kubeconfig
Cluster "kubernetes-the-hard-way" set.
User "system:kube-proxy" set.
Context "default" created.
Switched to context "default".
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/{kube-scheduler.pem,kube-scheduler-key.pem}
2021/08/27 12:52:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/kube-scheduler.kubeconfig
Cluster "kubernetes-the-hard-way" set.
User "system:kube-scheduler" set.
Context "default" created.
Switched to context "default".
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/{kubernetes.pem,kubernetes-key.pem}
[INFO] Creating /tmp/cfssl.iEzfftKA3/master/{service-account.pem,service-account-key.pem}
2021/08/27 12:52:55 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[INFO] Writing 127.0.0.1 to /tmp/cfssl.iEzfftKA3/nodes.2515bdb6e5e8/master
[INFO] Copying /tmp/cfssl.iEzfftKA3/master/ca.pem to /tmp/cfssl.iEzfftKA3/nodes.2515bdb6e5e8/ca.pem
[INFO] Creating /tmp/cfssl.iEzfftKA3/nodes.2515bdb6e5e8/{node.pem,node-key.pem}
[INFO] Copying /tmp/cfssl.iEzfftKA3/master/kube-proxy.kubeconfig to /tmp/cfssl.iEzfftKA3/nodes.2515bdb6e5e8/kube-proxy.kubeconfig
[INFO] Creating /tmp/cfssl.iEzfftKA3/nodes.2515bdb6e5e8/node.kubeconfig
Cluster "kubernetes-the-hard-way" set.
User "system:node:2515bdb6e5e8" set.
Context "default" created.
Switched to context "default".
[INFO] Base dir: /home/user/usernetes
[INFO] Installing /home/user/.config/systemd/user/u7s.target
[INFO] Installing /home/user/.config/systemd/user/u7s-master-with-etcd.target
[INFO] Installing /home/user/.config/systemd/user/u7s-rootlesskit.service
[INFO] Installing /home/user/.config/systemd/user/u7s-etcd.target
[INFO] Installing /home/user/.config/systemd/user/u7s-etcd.service
[INFO] Installing /home/user/.config/systemd/user/u7s-master.target
[INFO] Installing /home/user/.config/systemd/user/u7s-kube-apiserver.service
[INFO] Installing /home/user/.config/systemd/user/u7s-kube-controller-manager.service
[INFO] Installing /home/user/.config/systemd/user/u7s-kube-scheduler.service
[INFO] Installing /home/user/.config/systemd/user/u7s-node.target
[INFO] Installing /home/user/.config/systemd/user/u7s-containerd-fuse-overlayfs-grpc.service
[INFO] Installing /home/user/.config/systemd/user/u7s-kubelet-containerd.service
[INFO] Installing /home/user/.config/systemd/user/u7s-kube-proxy.service
[INFO] Starting u7s.target
+ systemctl --user -T enable u7s.target
Created symlink /home/user/.config/systemd/user/multi-user.target.wants/u7s.target → /home/user/.config/systemd/user/u7s.target.
+ systemctl --user -T start u7s.target
Enqueued anchor job 12 u7s.target/start.
Enqueued auxiliary job 32 u7s-kube-scheduler.service/start.
Enqueued auxiliary job 29 u7s-master.target/start.
Enqueued auxiliary job 30 u7s-kube-apiserver.service/start.
Enqueued auxiliary job 27 u7s-kube-proxy.service/start.
Enqueued auxiliary job 25 u7s-rootlesskit.service/start.
Enqueued auxiliary job 33 u7s-kube-controller-manager.service/start.
Enqueued auxiliary job 28 u7s-master-with-etcd.target/start.
Enqueued auxiliary job 14 u7s-containerd-fuse-overlayfs-grpc.service/start.
Enqueued auxiliary job 34 u7s-etcd.target/start.
Enqueued auxiliary job 13 u7s-node.target/start.
Enqueued auxiliary job 26 u7s-kubelet-containerd.service/start.
Enqueued auxiliary job 31 u7s-etcd.service/start.

real    0m5.706s
user    0m0.000s
sys     0m0.011s
+ systemctl --user --all --no-pager list-units 'u7s-*'
  UNIT                                       LOAD   ACTIVE SUB     DESCRIPTION
  u7s-containerd-fuse-overlayfs-grpc.service loaded active running Usernetes containerd-fuse-overlayfs-grpc service
  u7s-etcd.service                           loaded active running Usernetes etcd service
  u7s-kube-apiserver.service                 loaded active running Usernetes kube-apiserver service
  u7s-kube-controller-manager.service        loaded active running Usernetes kube-controller-manager service
  u7s-kube-proxy.service                     loaded active running Usernetes kube-proxy service
  u7s-kube-scheduler.service                 loaded active running Usernetes kube-scheduler service
  u7s-kubelet-containerd.service             loaded active running Usernetes kubelet service (containerd)
  u7s-rootlesskit.service                    loaded active running Usernetes RootlessKit service (containerd)
  u7s-etcd.target                            loaded active active  Usernetes target for etcd
  u7s-master-with-etcd.target                loaded active active  Usernetes target for Kubernetes master components (i…
  u7s-master.target                          loaded active active  Usernetes target for Kubernetes master components
  u7s-node.target                            loaded active active  Usernetes target for Kubernetes node components (con…

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
12 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
+ set +x
[INFO] Hint: `sudo loginctl enable-linger` to start user services automatically on the system start up.
[INFO] Hint: To enable addons including CoreDNS, run: kubectl apply -f /home/user/usernetes/manifests/*.yaml
[INFO] Hint: export KUBECONFIG=/home/user/.config/usernetes/master/admin-localhost.kubeconfig
$ uname -a
Linux N-20HJPF19TVVM 5.10.43.3-microsoft-standard-WSL2 #1 SMP Thu Jul 8 14:40:50 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
$ grep cgroup /proc/filesystems
nodev   cgroup
nodev   cgroup2
AkihiroSuda commented 3 years ago

Because the latest tag is set to v20210708.0, which still supports cgroup v1.