Is there a better way to deploy than to upload to nodes via SCP?

[figassis]

Depending on the local connection, it might take a while to deploy to multiple nodes. So it might be faster to generate the artifacts on each node. This also solves the issue of installing k8s-tew on a mac for example, and deploying to another OS like ubuntu.

If you run k8s-tew on a mac, it fails to generate the bash completion files because the generated executables do not run on a mac (kubectl for example). But running each of these directly on the nodes would solve this.

[darxkies]

Generating and downloading the files remotely would be possible even though probably not very resource friendly as the same files have to be downloaded multiple times.

I think the main issue here is actually kubectl and the other binaries that are used during the bootstrap process. It would suffice to just download the required binaries twice: once for the host system to generate the files / configure the nodes (in that case for Mac) and then again for Linux to be deployed. By using that distinction that should fix the issue. Isn't it?

[figassis]

The idea is that downloading is faster than uploading (most ansible scripts take advantage of this). Generating locally and uploading to N nodes can become a problem, But uploading the config to nodes and then running generate on each could allow us to then simply init the services on the deploy step. Even the bash completion files don't really need to be on the local machine.

On a mac, I made a few changes so that the bash completion files for kubectl and velero would be generated by running the local kubectl preinstalled on a mac and not the generated linux binary. But I should not need that because I won't use them on a mac anyway.

I'll try to implement this and see how it goes and maybe seek your help if it gets too deep.

[darxkies]

I understand now what you meant.

k8s-tew was born out of the need to instantiate HA Kubernetes clusters on bare metal run on premises. Meaning that the life-cycle is managed on premises. From that point of view the download rather than the upload of binaries is the main bottleneck.

Another advantage of the current approach is that the downloaded binaries together with all generated files (not only the bash scripts but also the yaml files, start script and so on) can be archived. That is also valid for container images, as they can be downloaded on the bootstrapping machine, uploaded and installed on the nodes. It did happen that once the cluster was instantiated, adding a new node later on, was problematic as binaries and container images went missing on the remote servers.

The question now is how can we achieve Mac support by retaining the current functionality.

The most convenient way would be to k8s-tew's ability to run commands when started as a daemon on the nodes. When k8s-tew starts as a supervisor for containerd and kubelet, it also runs various commands to turn off swapping, load kernel modules, install additional Ubuntu packages and so on. It could also generate the bash completion files. And you are right, those files do not need to be generated on the bootstrapping machine and they should be generated on the nodes. This way the "generate" sub command should work on Mac without problems.

[darxkies]

The deployment step uses meanwhile only kubectl to configure the cluster. The steps are defined in config.yaml.

So there are basically two options: 1) use a kubectl binary that can be executed on the bootstrapping machine, that would be also downloaded by k8s-tew 2) extend k8s-tew to bypass kubectl and communicate with the cluster directly. I did that for another project and it works well. Implementing that is not complicated as it might seem.

The second approach might be more elegant, but it would increase the size of the binary and the config file would have to be changed to contain only the yaml files that need to be executed.

[figassis]

I see your point, its a good idea to be able to archive the deployment files. I just ran into another issue where the deployment was failing because the node could not start the k8s-tew service. I was getting an "Exec Format" error from systemd. I realized it was because the executable was build on a mac and k8s-tew essentially copies itself to the nodes. So I added an OS check to download the executable if we're running on a mac.

I also think the second alternative is better. I also really want to ensure for example that k8s-tew can easily add new nodes to a running cluster by only deploying on new nodes, do avoid situations where the entire cluster can break with these issues.

In that case for example, these deployment problems would be detected on the new node and we could debug. Currently, a deployment issue ensures all nodes do not come up.

[darxkies]

IMHO the better solution is to add k8s-tew as a dependency just like containrd for example. It gets downloaded from github and done without having to deal with any OS checks.

Deploying only to new nodes sounds like a good idea for workers only. For controllers and storage nodes it does not work. When a controller or a storage it is added, the configuration of the other controllers and storage nodes changes as well and therefore they need to be updated.

Can you give some examples where adding new nodes might cause troubles to other nodes, when the deployment files are already achrived?

[figassis]

upgrading Kubernetes or any other binary is one way, as it required changing config.yaml and regenerating files. In my case I did that from my Mac and ended up deploying an incompatible k8s-tew to all nodes.

I only realized the issue by looking at the code, but yeah, I guess mostly the idea is to add worker nodes with little hiccups. Mostly I think there are a few deployment procedures that are unsupported and I think covering all is not the main goal so I guess documentation is one way to help avoid this. I want to help as much as possible to have this as production ready as possible.

[darxkies]

I have implemented most of the changes that we discussed above. When generating or deploying no external programs are invoked. Though, downloading itself from github is missing.

Are you still interested in a mac port?

[figassis]

Thanks! I’ll test sometime this weekend! I’ve been deploying from Ubuntu, but yes, still interested in a Mac environment, mostly to make it easier for a team to work on their current machines, which are macs. This is not urgent though and likely I’ll take a stab at it myself.

Cheers!

[darxkies]

Is this still relevant?