search

rootless-containers / usernetes

Kubernetes without the root privileges
https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2033-kubelet-in-userns-aka-rootless
Apache License 2.0
855 stars 58 forks source link

Method to setup u7s on multiple nodes #275

Closed cheungsuifai closed 10 months ago

cheungsuifai commented 1 year ago

Here is my solution to setup a u7s cluster with 2 nodes (one master and one worker):

Idea

  1. only kubelet and kube-proxy started on work node, while kube-apiserver, kube-controller and kube-scheduler not.
  2. kube-proxy is not runing on master(it can also set master as master+worker, but I want to seperate it to be more clearer)
  3. different Pod CIDR are allocated to different nodes which they belong to a larger CIDR
  4. etcd run along with master
  5. use containerd runtime
  6. use flannel cni
  7. generate certs to distribute them to all nodes manually

Design

Network info: master: master, 10.131.139.0 worker: k8s-node-80d1b085, 10.131.137.107

Service: master: u7s-master-with-etcd.target (if you want to make master as master and worker both, as it as u7s.target) worker: u7s-node.target

Pod CIDR: master: 10.0.42.0/24 worker: 10.0.43.0/24

Procedure

  1. Create rootless user on all nodes It's the same with the single node architecture, no detail operations provided.

  2. Setup certs

    
# generate certs
cfssldir=$(mktemp -d /tmp/cfssl.XXXXX)
cd common
./cfssl.sh --dir=$cfssldir --master=master --node=master,10.131.139.0 --node=k8s-node-80d1b085,10.131.137.107

copy certs on master node

config_dir="$HOME/.config"
rm -rf ${config_dir}/usernetes/{master,node} mkdir -p ${config_dir}/usernetes cp -r "${cfssldir}/master" ${config_dir}/usernetes/master cp -r "${cfssldir}/nodes.k8s-node-80d1b085" ${config_dir}/usernetes/node

sync certs to worker node

rsync --rsync-path=/usr/bin/rsync -r ${cfssldir}/* rootless@10.131.137.107:${cfssldir}

copy certs on worker node

set cfssldir var

config_dir="$HOME/.config" rm -rf ${config_dir}/usernetes/{master,node} mkdir -p ${config_dir}/usernetes cp -r "${cfssldir}/master" ${config_dir}/usernetes/master cp -r "${cfssldir}/nodes.k8s-node-80d1b085" ${config_dir}/usernetes/node


3. Setup master
use --cni to specify flannelcni
set Pod CIDR as 10.0.42.0/24 with --cidr
expose api-server port with --publish=0.0.0.0:6443:6443/tcp
only start master and etcd service
use pre-created certs with --wait-init-certs instead of auto creation.

./install.sh --cni=flannel --cidr=10.0.42.0/24 --publish=0.0.0.0:6443:6443/tcp --publish=0.0.0.0:2379:2379/tcp --publish=0.0.0.0:10250:10250/tcp --publish=0.0.0.0:8472:8472/udp --start u7s-master-with-etcd.target --wait-init-certs


4. Setup worker
use --cni to specify flannelcni
set Pod CIDR as 10.0.43.0/24 with --cidr
only start node service
use pre-created certs with --wait-init-certs instead of auto creation.

./install.sh --cni=flannel --cidr=10.0.43.0/24 --publish=0.0.0.0:10250:10250/tcp --publish=0.0.0.0:8472:8472/udp --start u7s-node.target --wait-init-certs



### Limitation
1. slirpnet network can not connected between different nodes
slirpnet->host network connection is ok, but host network->slirpnet is unreachable.
After investigation, it's pingable from a Pod in master to another Pod in worker.
But the path is very weired: 
After the package arrived on the destination host node, it did not forward to the namespace with slirpnet (10.131.137.107->10.0.43.0).
But forward to the namspace via flannel.

2. port exposure
nodeport is exposed on slirpnet network. But slirpnet is not exposed to external user.
It can publish the port to host with publish args of install.sh, but it seems it must stop the previous cluster first.
AkihiroSuda commented 1 year ago

port

You can expose NodePorts like this

./rootlessctl.sh add-ports 0.0.0.0:30080:30080/tcp

https://github.com/rootless-containers/usernetes#expose-netns-ports-to-the-host

cheungsuifai commented 1 year ago

port

You can expose NodePorts like this

./rootlessctl.sh add-ports 0.0.0.0:30080:30080/tcp

https://github.com/rootless-containers/usernetes#expose-netns-ports-to-the-host

Thank you for your remind. In the multiple nodes scenario, the above port exposure command should be executed on all nodes within the u7s cluster. Otherwise, the user will fail to access the nodeport service through the nodes which do not expose the desire port.

Ryadhmd commented 11 months ago

Hey there! 👋

I'm currently working on setting up a Kubernetes Cluster with two nodes using u7s. Here's a quick overview of my configuration:

I've generated the certificates using the following command:

./cfssl.sh --dir=$cfssldir --master=debian --node=debian,10.0.2.15 --node=worker,10.0.2.4

After that, I've placed the necessary folders in their respective locations and initiated the installation with:

./install.sh --cni=flannel --cidr=10.0.42.0/24 --publish=0.0.0.0:6443:6443/tcp --publish=0.0.0.0:2379:2379/tcp --publish=0.0.0.0:10250:10250/tcp --publish=0.0.0.0:8472:8472/udp --start u7s-master-with-etcd.target --wait-init-certs

However, the installation fails at various stages. I encounter the following issues:

  1. When running kubectl get nodes, it shows No resources found.
  2. There's a timeout waiting for condition on pods/coredns-** with the command: kubectl -n kube-system wait --for-condition=ready.

Upon investigation, I discovered:

I suspect that the problem is related to the option --start u7s-master-with-etcd.target, preventing pods from running on the master. I tried the installation without it, and the coredns pods started successfully. I was even able to launch deployments on the master node.

However, when I performed the installation on the worker node using:

./install.sh --cni=flannel --cidr=10.0.43.0/24 --publish=0.0.0.0:10250:10250/tcp --publish=0.0.0.0:8472:8472/udp --start u7s-node.target --wait-init-certs

Even though the installation claimed to be completed, the worker node doesn't appear when I run kubectl get nodes. As a result, new deployments only occur on the master.

I'm wondering if this issue could be related to the slirpnet network connectivity between different nodes, as mentioned by cheungsuifai.

Any insights or suggestions on resolving this issue would be greatly appreciated! Thank you! 🙏

AkihiroSuda commented 10 months ago

Please try this: