hobby-kube / guide

Kubernetes clusters for the hobbyist.
MIT License
5.57k stars 258 forks source link

Maintenance burden #51

Closed zladuric closed 5 years ago

zladuric commented 6 years ago

I am trying to estimate how resource-intensive is to maintain this cluster. I'm pretty decent with linux, shells, networking, the concept, I could even fall back to iptables without a hiccup. But my current job is relatively far from devops.

So setting this cluster up would take several hours, up to a day, I assume. That I can afford. But I wonder how up-to-date is it important to be, and furthermore, how frequent are updates to various parts of the project. By using Google's containers, I have a feeling every time I log in to my computer there's an update to gcloud and kubectl packages.

I would maintain this cluster purely for personal reasons - playing with it a bit, maybe learning etc. But if I need to spend a lot of time just maintaining this, then I'm probably better off using Scaleway or Hetzner directly.

Can anybody share any insights into this topic? How often do you have to do updates etc to this system?

aleks-mariusz commented 6 years ago

It's no secret that Kubernetes (and anything devops/cloudy in general) is a fast-moving set of technologies. I too am in a similar position, quite able with linux/shell/networking concepts (i'm very well seasoned at the risk of sounding not-very-humble). I'm currently attempting to lead the containerization strategy at the firm i recently started in so for me it's very important (not to mention very interesting) to get familiar with how things are done.

My advice to you would be to play with mini-kube. It is an all-in-one 'cluster-in-a-box' setup that will get you started in minutes not hours. You can do that for a week or two to get familiar with the general kubernetes concepts, at least if you are totally new. I should have done that probably but i did a lot of reading instead, and ended up a few weeks ago jumping straight into trying to set up a 3x master HA set up with integrated storage (for persistent volumes). Though I've learned a LOT doing that, the process was anything but easy, it was however quite rewarding. My current struggle is getting centralized logging working, mainly because at some point (or maybe just specific to the distro i use, CentOS 7), docker doesn't log to json-files, and unfortunately all the fluentd/file-bit/etc examples i've come across kind of expect that, with very little supporting systemd's journald services that docker/kubelet/etc log to.

One thing to keep in mind that i've learned is to pay attention to how actively supported any projects you're considering adopting are. If they haven't been touched in a year, that's a bit of a red flag. As you are probably aware of how fast it moves, keeping it up to date isn't something you can completely forget about. If you are planning to be near the cutting-edge of this technology then it's good to be aware of how the landscape is slowly changing (such as the general move toward systemd i'm suspecting down the line). If you want to be on the very cutting edge (testing nightly builds and such) then updating is going to be a fact of life for you.

So to answer your question, how often to update? keep an eye on it every few weeks probably. The documentation is quite good, set up a test cluster in addition to your main cluster, script the re-creation of the test cluster and have it mirror your main cluster, testing any new technology in the test cluster. That's what i'm doing, with a lab cluster i rebuild at least once or twice a week with new lessons learned. The solidified ones will end up in my main cluster (mainly for dev work) and eventually i'm hoping to have a prod cluster in AWS that will have zero one-off things and all deployed via automation (such as the gitops mindset that things like Helm and Weave Flux gives you).

rafi commented 5 years ago

First and foremost, to delve into these technologies you better have a passion for cluster computing and containerization in general. It sounds like you do, even though you're not in an official "devops" title. I always loved linux & devops and finally changed my title 2 years ago. I started learning Kubernetes half a year ago, and currently I'm managing a few clusters at work and one for my pet project.

I highly recommend minikube as a starting point as @aleks-mariusz suggested. I used kubespray quite a lot, but I love this guide, and used it for my personal project to practice Terraform and spin up a Kubernetes cluster quickly, without Ansible.

Regarding upgrades and maintenance, today I'd rather simply spin up a new upgraded cluster and migrate the workload there instead of trying to upgrade the "old" cluster in-place. Even changing master roles or etcd nodes might get complicated with certain deployment tools, so as long as I keep a well-documented setup instructions and commit all the infrastructure manifests, I can spin up a new cluster in no time.

Finally, there's nothing wrong with running older versions a while. I have a variety of kubernetes clusters running from versions 1.10.x-1.12.x. Just remember never to use the latest image tags :)

zladuric commented 5 years ago

Thanks both for very useful insight. Especially the tip to simply set up new clusters on a certain schedule (e.g. Every six weeks or so) and migrate the workload three. In the mean time, I did get the cluster up and running in hetzner VMs, wasn't that scary at all. I kind of neglected it in the mean time due to extremely busy schedule, but now I wanna get back to it.

Thanks again to both for the feedback.

pstadler commented 5 years ago

Forgot about this issue.

I generally recommend to treat clusters as immutable (besides scaling, but this is an entirely different topic).

Good to know that other people share this opinion 🍻