search

kubesphere / kubekey

Install Kubernetes/K3s only, both Kubernetes/K3s and KubeSphere, and related cloud-native add-ons, it supports all-in-one, multi-node, and HA 🔥 ⎈ 🐳
https://kubesphere.io
Apache License 2.0
2.36k stars 549 forks source link

[KubeKey v2.0.0]Support for OS that has read-only /usr/* paths #669

Open TheApeMachine opened 3 years ago

TheApeMachine commented 3 years ago

Your current KubeKey version

v1.1.1

Describe this feature

Given my good experiences with KubeSphere I wanted to start using KubeKey to provision more bare-bones Kubernetes clusters as well. The thing is that I run Flatcar Linux as the base OS, which has everything under /usr/** as read-only. I forked the repo to have a closer look, because it seems like a pretty easy thing to change. I see most of the paths are constants at the moment, and as quick as I have looked over it, I could not see an immediate way to use another bound cli flag and pass it all the way through the method path that would need to change. Before I look further into it, is this something that you would want to support in the first place?

Describe the solution you'd like

I think another flag that allows you to set the bin path to /opt/bin should be all that is needed and in my opinion stays pretty close to the overall design of KubeKey. Only one or two paths would have to be lifted out of the constants, and current behavior can still remain the default.

./kk create cluster [--with-kubernetes version] [--with-kubesphere version] [--with-bin-path /opt/bin]

Something like the above?

Additional information

Honestly I am not sure if this is too niche, or maybe just at the moment. Personally I see a lot of mentions about Flatcar Linux, even de official Kubernetes documentation makes mention of it. Besides that I think it could become more and more of a thing as there are more OS's aiming for very secure defaults (think Talos for instance).

And I honestly don't know how much work it is. Doing a quick search through the code, there where quite a few mentions of the /usr/ path.

FeynmanZhou commented 3 years ago

It sounds like it makes sense for us to support this feature. I am curious if Kubeadm or Kubespray support the OS that has read-only /usr/* paths?

@pixiake @24sama @kubesphere/sig-installation Please also help to evaluate this kind of scenario.

pixiake commented 3 years ago

I think this should be a reasonable demand. If it only supports the replacement of bin path, I think the next version can support it. But for Flatcar Linux, I think we should do a comprehensive test.

FeynmanZhou commented 3 years ago

@TheApeMachine Would you like to help us to support it or discuss a proposal?

TheApeMachine commented 3 years ago

If you wanted me to have a go at the code, it would probably take me a while. I don't think the code itself is super hard, but I know nothing about the build process or anything and I suspect to test it well enough would take up the bulk of the time. I imagine there are people on KubeSphere's end that would have the code turned around in almost no time, if you know the code this should be pretty easy. However, I would try if you have no time for it. And I am definitely willing to help test it, I have a pretty quick path to setting up Flatcar Based Kubernetes clusters at the moment, and I have a Dell r710 free here that would be provisioned within a few minutes. Using shell scripts :p For the rest I am not very familiar how things go in the open-source world. I've just been kinda struggling managing the shell scripts and I remember how easy it all was deploying KubeSphere, so I see a lot of merrit in KubeKey as a general tool. Kubeadm does support it, it is what I am using now. Kubespray I do not know.

mister2d commented 3 years ago

It may be worth leveraging some of the implementation that Typhoon has used for Flatcar Linux k8s deployments.

TheApeMachine commented 3 years ago

Yes, I did not even know that was still an active project, since on their website they seem to be pushing towards Lokomotive (https://kinvolk.io/lokomotive-kubernetes/). Although I guess that is more a distribution? I tried to use it, a little too complex for me now, so I just have a very big systemd unit now that installs everything using kubeadm. It is relatively similar to the KubeSphere setup I guess, though I do not use NGINX. After kubeadm for the cluster I deploy metallb and istio (I tried porterlb, but could not make it work, because I am a beginner). Ah and I use weavenet instead of calico, because I wanted to try it to see if it would work better with weave scope (it does a little). I believe that is it, the only thing left I do manually after that is install ArgoCD and it pulls the rest of the setup out of our company's git repos, which works surprisingly well.

TheApeMachine commented 3 years ago

I'd be happy to share the working configs though if that would help? And again, if you are interested but crushed for time I would even try to implement it, but I know for sure that would take me a while.

mister2d commented 3 years ago

Yes, I did not even know that was still an active project, since on their website they seem to be pushing towards Lokomotive (https://kinvolk.io/lokomotive-kubernetes/). Although I guess that is more a distribution? I tried to use it, a little too complex for me now, so I just have a very big systemd unit now that installs everything using kubeadm. It is relatively similar to the KubeSphere setup I guess, though I do not use NGINX. After kubeadm for the cluster I deploy metallb and istio (I tried porterlb, but could not make it work, because I am a beginner). Ah and I use weavenet instead of calico, because I wanted to try it to see if it would work better with weave scope (it does a little). I believe that is it, the only thing left I do manually after that is install ArgoCD and it pulls the rest of the setup out of our company's git repos, which works surprisingly well.

Typhoon is definitely active and was just updated to support Kubernetes 1.22.2. It doesn't have to change all that much since it's just Terraform code.

The reference to Flatcar Linux (that Lokomotive site) is just there to further extend your Terraform code if needed. Typhoon uses Flatcar Linux as the distribution for production Kubernetes deployments.

TheApeMachine commented 3 years ago

I see now yes :) Thank you. It is using that same matchbox deployment technique I had seen in other tutorials, which I tried and kind of worked, but then I ran into my lack of knowledge about terraform. I did make another setup which also uses PXE deployment and just exposes a darkhttp server with the ignition json for download and that worked quite well, but that was just installing the Flatcar OS, no kubernetes yet because I wasn't at that point yet. This I have wrapped up in docker/compose setup for convenience, though I am sure terraform is much more useful, I just have too many things to learn to also take that on.

pixiake commented 3 years ago

@all-contributors please add @TheApeMachine for ideas

allcontributors[bot] commented 3 years ago

@pixiake

I've put up a pull request to add @TheApeMachine! :tada: