rootless-containers / usernetes

Kubernetes without the root privileges
https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2033-kubelet-in-userns-aka-rootless
Apache License 2.0
855 stars 58 forks source link

why overlay snapshotter is feasible with u7s? #269

Closed cheungsuifai closed 1 year ago

cheungsuifai commented 1 year ago

Hello, I switched the containerd snapshotter from "fuse-overlay" to "overlay" by modified containerd.sh as below: (proxy_plugins was deleted and snapshotter changed from "fuse-overlay" to "overlay")

#!/bin/bash
# needs to be called inside the namespaces
export U7S_BASE_DIR=$(realpath $(dirname $0)/..)
source $U7S_BASE_DIR/common/common.inc.sh

mkdir -p $XDG_RUNTIME_DIR/usernetes
cat >$XDG_RUNTIME_DIR/usernetes/containerd.toml <<EOF
version = 2
root = "$XDG_DATA_HOME/usernetes/containerd"
state = "$XDG_RUNTIME_DIR/usernetes/containerd"
[grpc]
  address = "$XDG_RUNTIME_DIR/usernetes/containerd/containerd.sock"
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "mirrors.sangfor.com/google_containers/pause:3.8"
    disable_cgroup = false
    disable_apparmor = true
    restrict_oom_score_adj = true
    disable_hugetlb_controller = true
    [plugins."io.containerd.grpc.v1.cri".containerd]
      snapshotter = "overlayfs"
      default_runtime_name = "crun"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun.options]
            BinaryName = "crun"
    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "$XDG_RUNTIME_DIR/usernetes/certs.d"
EOF

exec containerd -c $XDG_RUNTIME_DIR/usernetes/containerd.toml $@

After that, I deploy the u7s cluster with install.sh and all the components were running. it seems all ok.

And now i wondering why fuse-overlay is needed in u7s?

AkihiroSuda commented 1 year ago

FUSE-overlayfs is just for supporting old kernel.

cheungsuifai commented 1 year ago

FUSE-overlayfs is just for supporting old kernel.

Thank you for your reply.

According to user namespace man: https://man7.org/linux/man-pages/man7/user_namespaces.7.html it is ok to mount overlay with kernel >= 5.11

But my system is below 5.11: uname -a Linux master 4.18.0-372.9.1.15.po1.x86_64 https://github.com/rootless-containers/usernetes/pull/1 SMP Mon Jul 4 13:53:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

AkihiroSuda commented 1 year ago

What’s your disto? They might have been using Ubuntu’s kernel patch to enable overlayfs in userNS with old kernel releases.

cheungsuifai commented 1 year ago

What’s your disto? They might have been using Ubuntu’s kernel patch to enable overlayfs in userNS with old kernel releases.

actually, it is a customized disto, but it seems based on CentOS. But anyway, I think the problem is solved. I can just upgrade the kernel to desired version to adapt overlay. Thank you very much.