nestybox / sysbox

An open-source, next-generation "runc" that empowers rootless containers to run workloads such as Systemd, Docker, Kubernetes, just like VMs.
Apache License 2.0
2.78k stars 152 forks source link

Sysbox-k8s-deploy unable to handle '--cni-bin-dir' kubelet parameter #563

Closed rodnymolina closed 2 years ago

rodnymolina commented 2 years ago

Sysbox-k8s-deploy installer is not currently taking into account kubelet's 'cni-bin-dir' configuration flag. In consequence, kubelet complains about being unable to operate the cluster:

CNI network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied

Kubelet is launched with the following flags (notice the --cni-bin-dir flag):

root      119974       1  3 00:31 ?        00:02:05 /home/kubernetes/bin/kubelet --v=2 --experimental-check-node-capabilities-before-mount=true --cloud-provider=gce --experimental-mounter-path=/home/kubernetes/containerized_mounter/mounter --cert-dir=/var/lib/kubelet/pki/ --kubeconfig=/var/lib/kubelet/kubeconfig --cni-bin-dir=/home/kubernetes/bin --image-pull-progress-deadline=5m --max-pods=110 --non-masquerade-cidr=0.0.0.0/0 --network-plugin=kubenet --volume-plugin-dir=/home/kubernetes/flexvolume --node-status-max-images=25 --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --runtime-cgroups=/system.slice/containerd.service --registry-qps=10 --registry-burst=20 --config /home/kubernetes/kubelet-config.yaml --pod-sysctls=net.core.somaxconn=1024,net.ipv4.conf.all.accept_redirects=0,net.ipv4.conf.all.forwarding=1,net.ipv4.conf.all.route_localnet=1,net.ipv4.conf.default.forwarding=1,net.ipv4.ip_forward=1,net.ipv4.tcp_fin_timeout=60,net.ipv4.tcp_keepalive_intvl=60,net.ipv4.tcp_keepalive_probes=5,net.ipv4.tcp_keepalive_time=300,net.ipv4.tcp_rmem=4096 87380 6291456,net.ipv4.tcp_syn_retries=6,net.ipv4.tcp_tw_reuse=0,net.ipv4.tcp_wmem=4096 16384 4194304,net.ipv4.udp_rmem_min=4096,net.ipv4.udp_wmem_min=4096,net.ipv6.conf.all.disable_ipv6=1,net.ipv6.conf.default.accept_ra=0,net.ipv6.conf.default.disable_ipv6=1,net.netfilter.nf_conntrack_generic_timeout=600,net.netfilter.nf_conntrack_tcp_be_liberal=1,net.netfilter.nf_conntrack_tcp_timeout_close_wait=3600,net.netfilter.nf_conntrack_tcp_timeout_established=86400

However, this config attribute is not taken written into cri-o configuration during sysbox-k8s-deploy installation:

[crio]
  storage_driver = "overlay"
 storage_option = ["overlay.mountopt=metacopy=on"]

[crio.image]

[crio.network]

[crio.runtime]
    conmon_cgroup = "pod"
    cgroup_manager = "cgroupfs"
    default_capabilities = ["CHOWN", "DAC_OVERRIDE", "FSETID", "FOWNER", "SETUID", "SETGID", "SETPCAP", "SETFCAP", "NET_BIND_SERVICE", "KILL", "AUDIT_WRITE", "NET_RAW", "SYS_CHROOT", "MKNOD"]
    pids_limit = 16384
    selinux = false

[crio.runtime.runtimes]
   [crio.runtime.runtimes.sysbox-runc]
       allowed_annotations = ["io.kubernetes.cri-o.userns-mode"]
       runtime_path = "/usr/bin/sysbox-runc"
       runtime_type = "oci"
rodnymolina commented 2 years ago

Fix went into latest sysbox-k8s-deploy images.

rodnymolina commented 2 years ago

Closing.