Not exactly a bug, it's a question regarding Security.

[kchsrinadh]

Before creating an issue, make sure you've checked the following:

• [X] You are running the latest released version of k0s
• [X] Make sure you've searched for existing issues, both open and closed
• [X] Make sure you've searched for PRs too, a fix might've been merged already
• [X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Darwin 22.5.0 Darwin Kernel Version 22.5.0
RELEASE_ARM64_T6000 arm64

Version

1.21.4

Sysinfo

No response

What happened?

Sorry to report this as bug. I couldn't find a QA or general session in creating GitHub issue.

I am new to k8s and I am trying to understand how to setup cluster using k0sctl. As I feel k0s much beginner friendly approach.

I find it interesting to use SSH to bring the nodes up with right configuration instead of me manually setting them.
I am referring to this session in docs : https://docs.k0sproject.io/v1.21.0+k0s.0/k0sctl-install/#2-configure-the-cluster I am assuming behind the scenes somewhere k0sctl uses tokens for node registration.

But doesn't it create a security issue with ssh ? If a control plane is compromised will that mean all the ssh keys to worker nodes will be compromised ?? Or am I missing something here ?

Just curios to know how k0s handles or does it have any security measures already in place for my following concerns SSH Access: The tool needs SSH access to the nodes. If the SSH keys or credentials are mishandled or stored insecurely, unauthorized users could gain access to your nodes. Permission Level: Typically, administrative or root-level access is required to perform many of the operations. If an attacker gains control of the tool or the system where it's run, they could potentially execute arbitrary commands on your nodes. Man-in-the-Middle Attacks: If SSH is not configured to use strong encryption and authentication, there could be potential for man-in-the-middle attacks, where an attacker intercepts and possibly alters the communication between the tool and the nodes. Lack of Network Segmentation: If SSH is open to the world and not restricted to specific IP addresses or networks, it increases the attack surface. Automation Risks: Automated tools that perform complex tasks can sometimes introduce unexpected security holes, depending on how they are configured and used.

Are above concerns handled by k0sctl behind the scenes ?

Steps to reproduce

1 Following the docs https://docs.k0sproject.io/v1.21.0+k0s.0/k0sctl-install/#2-configure-the-cluster

2 this is related to following section in the above docs https://docs.k0sproject.io/v1.21.0+k0s.0/k0sctl-install/#2-configure-the-cluster

Expected behavior

No response

Actual behavior

No response

Screenshots and logs

No response

Additional context

No response

[kke]

I am assuming behind the scenes somewhere k0sctl uses tokens for node registration.

K0sctl uses k0s's own mechanism to create join tokens for additional nodes.

1. For each additional node k0sctl generates a join token using k0s token create --role=XX --expiry 10m on the first controller node.
2. This token is copied to the other node's /etc/k0s/k0stoken.
3. K0s is installed and started on the other node, which will use the join token for authenticating the node to the k0s/k8s cluster.
4. Once the node has joined the cluster, k0sctl uses k0s token invalidate to invalidate the token and removes the key from /etc/k0s/k0stoken as it is not needed anymore. This will also be done if the installation fails to avoid leaving behind unused tokens.

If a control plane is compromised, will that mean all the SSH keys to worker nodes will be compromised ??

The nodes in your cluster don't require SSH access to each other. K0sctl is primarily used from your workstation or a deployment system and isn't actually installed on the cluster itself. It's intended for use by individuals or systems with administrative-level SSH access to the nodes. K0s doesn't need or utilize k0sctl and you don't need k0sctl to manage k0s – it's a separate optional utility for simplified k0s deployments.

SSH Access: The tool needs SSH access to the nodes. If the SSH keys or credentials are mishandled or stored insecurely, unauthorized users could gain access to your nodes.

This principle applies universally to systems with any form of authentication. Avoid mishandling or insecure storage of credentials.

Permission Level: Typically, administrative or root-level access is required to perform many of the operations. If an attacker gains control of the tool or the system where it's run, they could potentially execute arbitrary commands on your nodes.

Using passphrase-encrypted SSH keys adds an extra layer of protection against potential breach if your local system were to be compromised.

Man-in-the-Middle Attacks: If SSH is not configured to use strong encryption and authentication, there could be potential for man-in-the-middle attacks, where an attacker intercepts and possibly alters the communication between the tool and the nodes. Lack of Network Segmentation: If SSH is open to the world and not restricted to specific IP addresses or networks, it increases the attack surface. Automation Risks: Automated tools that perform complex tasks can sometimes introduce unexpected security holes, depending on how they are configured and used. Are the above concerns handled by k0sctl behind the scenes?

Configuring SSH and network security falls outside k0sctl's scope.

[kchsrinadh]

Hi

@twz123 thank you for assigning the right labels. @kke Thank you very much for clarifications. It explains a lot clearly of whats happening in background.

Once the node has joined the cluster, k0sctl uses k0s token invalidate to invalidate the token and removes the key from /etc/k0s/k0stoken as it is not needed anymore. This will also be done if the installation fails to avoid leaving behind unused tokens.

I just have 1 question left. So if the token is invalidated. Say in future, If the node changed its Ip address due to some reason. Will that node still be able to register itself with the control plane. I am specifically looking for node registration for nodes that can have dynamic IP.

[kke]

After the initial join, the workers continue to communicate with the cluster using x509 client certificates through the kube api. The controllers do not directly communicate with the workers.

In kubelet there's:

--hostname-override string   If non-empty, will use this string as identification instead of the actual hostname. If --cloud-provider is set, the cloud provider determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used).

K0sctl already validates that every node has a unique hostname, so I think it may just work even without --cloud-provider, but I'm not sure, as k0sctl does set the --node-ip when joining workers:

--node-ip string IP address (or comma-separated dual-stack IP addresses) of the node. If unset, kubelet will use the node's default IPv4 address, if any, or its default IPv6 address if it has no IPv4 addresses. You can pass '::' to make it prefer the default IPv6 address rather than the default IPv4 address.

Anyway, just recording api communication and reusing an existing worker's IP won't allow a rogue node to communicate with the control plane, as it needs the client cert private key from the real worker.