Kubernetes?

[nikola-rados]

Questions

At PCIC we are in the middle of building some new servers that will be capable of running Kubernetes. I feel like there is potential to have birdhouse run on k8s by taking the same infrastructure that is in place for docker and translating it. There are tools like kompose and kustomize that have the potential to help with this process. That being said I imagine there would be a fair amount of work to get something like this operational.

I just wanted to spark this conversation and see what people think. I'm not sure if this is something the wider birdhouse audience would be interested in but at PCIC we are eager to try it out.

[github-actions[bot]]

Thanks for submitting an issue. Please make sure to provide enough details for us to be able to replicate your issue or understand your question.

[matprov]

Hi @nikola-rados, having the workers running on k8s nodes is one of our desire. The workers need to scale up/down based on the demand, while the rest of the platform might stay on a simple docker-compose stack, for now.

We have a PoC on our side that is able to scale workers based on the jobs we send to the queue. That being said, it's experimental and not thoroughly tested. It is currently not part of our main objectives for the next few months.

Let me know if you are interested in seeing what has already been done on this side.

[dbyrns]

Hi @nikola-rados, as for running the whole stack on k8s I would chime in that it's very interesting and we are not against it. However, as you said, it would require a lot of work and an eventual PR would need to be back-compatible so that we can still start the stack using compose if required. We have invested a lot on a stable infrastructure based on compose, including a complex templating script, activatable optional components and IAC testing stack. Sadly, we cannot afford right now to support any other techno and to accept contributions we would need to be sure that all those features still work with them.

So if you guys want to contribute, I hope that I didn't discourage you to do so. I was just letting you know what to expect.

[tlvu]

Hi @nikola-rados, I am along the same line as @matprov and @dbyrns above. Kubernetes is definitively in our roadmap but we need something backward-compat with the current compose stack, for now.

The current plugable design of the compose stack (by activating new components committed directly in the same repo or from an external repo) means there is not one single docker-compose.yml but multiples fragments so I wonder if kompose is compatible with that. If it is, then one transparent way to try k8s is to run kompose on the same list of fragments. Then the same stack can be run both on docker-compose and k8s.

The kustomize idea seems to be the equivalent of the multiple compose fragments but for k8s, so very interesting if the source config format is k8s and not docker-compose.

I am also interested in your recipe to spin up a k8s cluster. If you have ongoing issue or PR somewhere, feel free to tag me.

[nikola-rados]

Thank you all for your comments! I appreciate the feedback.

The way I imagine building this k8s stack would be alongside the docker-compose stack while factoring out common configuration items into some general config directory. This would keep all the current functionality in place while building a separate feature set. That is just what comes to mind first, do any of you have ideas or comments on how best to integrate k8s?

In any case I feel the best way to go about this would be to fork birdhouse-deploy into the pacificclimate organization. That would give us a bit more control over things and not clutter up the main repo with experimental issues/PRs.

Also just as a note, I don't imagine we will be starting on this until we have a better understanding of how to use k8s. So we will migrate some of our tools over first, before attempting the same with birdhouse-deploy.

[matprov]

@nikola-rados

Two challenges comes in mind when dealing with k8s, regarding the worker nodes. First one is the configs management; how are they defined, updated, and securely distributed across your running nodes. Second challenge is the volume management; data retention quotas, ephemeral vs persistent storage, storage managed in k8s vs elsewhere.

To get a head about these topics, forking the birdhouse-deploy repo under your organisation is a good idea. Knowing how each component is currently configured on birdhouse-deploy is a must before building the stack on k8s. Start small, a component/concept (eg: autoscaling, volumes, configs, lifecycle) at a time in order to understand how it fits on your k8s infrastructure. Trying to migrate the whole stack to k8s at the same time (eg: using kompose, who's supposed to make your life easier) simply won't work, and knowing the cause will be extremely difficult.

So yes, factoring out one piece at a time is the best way to do. Workers have been identified as the most valuable stuff to put on k8s first, to master autoscaling.

For any help on this kind of stuff don't hesitate to ping me.

[tlvu]

@nikola-rados

Also test upgrade procedure. From what I've read, updating a k8s cluster is a bit more involved than just updating docker and docker-compose binaries.

Forking the birdhouse-deploy repo is a good idea. In fact Ouranos and CRIM already have our own fork to have finer control to what goes into our respective production. You can also leverage a branch in your fork as sandboxing.

The way I imagine building this k8s stack would be alongside the docker-compose stack while factoring out common configuration items into some general config directory. This would keep all the current functionality in place while building a separate feature set. That is just what comes to mind first, do any of you have ideas or comments on how best to integrate k8s?

Yes I think that's a right approach. Maybe start with a small component instead, I'd suggest generic_bird component that has only 2 containers, one of which is a DB (will force you to think about persistent storage and how the container moves between hosts without losing data).

That generic_bird is a self-contained component. If you can generate k8s config from what's where (or a slight modification of what's there but still compat with docker-compose) it's a very good start.

More or less how we currently instantiate config files from template, if we can instantiate k8s config, then the same stack can support both deployments to docker-compose and k8s.

That generic_bird matches more or less how you would "integrate" your birds into the stack. Have a quick look at the brief docs of generic_bird here https://github.com/bird-house/birdhouse-deploy/tree/master/birdhouse/optional-components#a-generic-bird-wps-service

[tlvu]

FYI some link about Gitlab Kubernetes integration from a presenter (Zachary Schoenberger from UVic) at Canarie conference.

https://gitlab.com/calincs/infrastructure/configuration/-/blob/master/docs/gitlab_integration.md https://gitlab.com/calincs/infrastructure/lincs-rs-custom-app/-/blob/master/.gitlab-ci.yml

Might be of interest to both CRIM (using Gitlab) and PCIC (UVic).

[matprov]

Hi @nikola-rados, haven't tried it but this might be of interest https://github.com/Pablo-CC/birdhouse-helm.