search

xmidt-org / xmidt

Highly scalable pipes for communicating with devices all over the place.
Apache License 2.0
17 stars 19 forks source link

Added k8s deployment #14

Closed florianbachmann closed 4 years ago

florianbachmann commented 4 years ago

Dear Ladies and Gentleman,

this Pull Request tries to add support k8s for xmidt. It is WIP and hopefully prevents someone from doing duplicated work. In the best case, this Pull Request should start a a discussion about possible scaling implications when deploying xmidt with k8s (like we are aiming to do it here in our DTAG k8s cluster). ​ The added k8s deployment is heavily based on the provided docker-compose deployment. A helm chart is added to express dependencies to Consul and Prometheus(wip). Most config files (from ./deploy/docker-compose/docFiles ) are added to ConfigMaps in their respective files originated at ./deploy/kubernetes/xmidt-cloud/templates. As k8s needs access to a docker image registry it would be nice to add public repositories on hub.docker.com for each service (petasos, talaria, etc.). Take a look at ./deploy/kubernetes/xmidt-cloud/values.yaml. It would be even better to automate this with a CI process. Maybe Github Actions?

Scaling:

Right now, only one talaria instance is deployed, due implications with service discovery. The thing is, each talaria instance registers itself in Consul with a hardcoded address on which it is reachable by others. Excerpt from talaria config (./deploy/kubernetes/xmidt-cloud/templates/talaria.yaml)

        registrations:
          -
            id: "talaria"
            name: "talaria"
            tags:
              - "dev"
              - "docker"
              - "stage=dev"
              - "flavor=docker"
            address: "http://talaria"
            scheme: "http"
            port: 6200
            checks:
              -
                checkID: "talaria:http"
                http: "http://talaria:6201/health"
                interval: "30s"
                deregisterCriticalServiceAfter: "70s"

This doesn't play well with K8s scaling feature (replicas), as traffic is normally load balance across a ReplicaSet. In our case it is important to route requests to the correct talaria instance as only one specific talaria instance holds the websocket connection to a device. One solution could be to use k8s "StateFulsets" promise about "Stable, unique network identifiers." https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id, but we are not yet sure how to pass this information down to talarias config. This needs some further investigation on my side. Or maybe you have a better solution how to tackle this. I don't have too much insights to the other services, so there might be scaling implications as well. Even though for experimentation with a k8s deployment one talaria instance seems to be enough. ​

TL;DR:

Adds k8s support to xmidt. But we

best regards, Florian Bachmann and Matthias Nösner ​

CLAassistant commented 4 years ago

CLA assistant check
All committers have signed the CLA.

kcajmagic commented 4 years ago

Awesome, thanks for submitting a PR on K8s. It has been something that I have been wanting to get working for a while now.

Overall this PR looks good, and I absolutely love that you added the README.md file for for it.

Scaling in general for XMiDT inside of K8s is a hard problem, that I haven't found a good solution for because an outside client needs to be able to talk to any talaria server at any given time. It appears that StatefulSets solves this problem. As for the other services, using round robing between the pods will be totally fine.

The config file can be overwritten with environment variables. For example, if you want to change the primary address port of talaria to 7777 you can set the environment variable TALARIA_PRIMARY_ADDRESS=":7777"

kcajmagic commented 4 years ago

@florianbachmann @Equanox I think each service should have its own helm chart so consumers can pick and choose which services they want to stand up. Do you have any opinions on how this should move forward?

Equanox commented 4 years ago

@kcajmagic as xmidt is separated into multi repos, sure it also makes sense to have a helm chart for each service and have this helm chart here as a example to put everything together. I can provide pull requests for each service separately.

This opens up another question: The best way to depend on other helm charts, is to consume them from a chart_repository. Guess this will take some extra effort and we need to decide if we want to use the official one or use a self hosted one . Think both variants need some kind of a automated CI/CD process.

Though, as a starting point we can use a script o automate packaging of all xmidt helm charts. Into this one.

Equanox commented 4 years ago

@kcajmagic now this helm chart depends on helm charts from these PR's:

I was able to make requests to a simulated device using a XMIDT-cloud deployment based on this chart.

kcajmagic commented 4 years ago

@Equanox I think this looks cleaner. Before I merge this I would like to get some discussion on the future of xmidt and kubernetes. I have created a new topic on the new disscussion site of xmidt.

kcajmagic commented 4 years ago

@Equanox I got the xmidt cluster working but I had to change apiVersion: v2 in Chart.yaml to v1 for each service. This might be a local issue, any ideas?

Can you also rebase the PR. This provides a good foundation so we can make the upgrades necessary to make it scale later.

Equanox commented 4 years ago

Did you use helm3? I think helm2 requires apiVersion: v1. The chart skeletons were created using helm create chart_name. This adds apiVersion: v2 to Chart.yaml.

According to this discussion https://github.com/helm/helm/issues/5907#issuecomment-516063552 apiVersion: v2 seems to be the correct choice for helm3.

kcajmagic commented 4 years ago

upgrading to helm 3 worked