open kubernetes facility

[nikhiljha]

It's fun to imagine how much simpler OCF infra could be if everything was just kubernetes. Assume all of this is backed by a ceph node.

Normal Difficulty

• munin and prometheus
• CS:GO server (does anyone actually use this?)
• Jenkins
• IRC server on kubernetes

Hard Difficulty

• move web hosting to kubernetes (redundant apache pods instead of a VM with apache)
  • no more Haproxy, just run traefik in kubernetes
  • no more duplicated state (ocf/dns ocf/puppet to point a new domain name to kubernetes)
• move HPC to kubernetes (JupyterHub)
• move postgres to kubernetes
  • if we need mysql we can move that too

Insane Difficulty

• SSH pod (tsunami) running in kubernetes, exposes a fake/lightweight environment to move files around and run scripts

Impossible

• move desktops to kubernetes (hydra is now a kubernetes node)
• mail server in kubernetes
  • this is completely insane because touching the mail server unless absolutely necessary is completely insane

[jvperrin]

I'll add some notes on a few of these:

• IRC server on kubernetes I think would be pretty difficult because it's stateful in terms of connections. Maybe it would work with multiple instances federated together or something? Idk
• Jenkins I think would be more difficult than you expect because it's spawning containers of its own for some builds (say for building services). Maybe it could be done with docker-in-docker or something, but I don't think it would be a straight port over, increasing the difficulty here
• apache2 I think would actually be reasonably easy, it's the NFS access that it would need that I believe would make it more difficult (this is also #25)
• mail server in kubernetes was actually proposed back when we were on mesos/marathon (rt#5903) and I think it would actually be pretty straightforward since for the most part it's stateless and email is quite lenient on retries.

[ja5087]

Jenkins can actually spawn build agents dynamically on k8s so I think it's entirely possible (EDIT: nvm I misread, maybe something like https://link.medium.com/PSw4d5IQCab). I think Ceph will enable a lot of things that were previously blocked on NFS, but I still think databases/IRC/stateful workloads will be a while until we're comfortable running them on the kube.

[nikhiljha]

progress on this: we've figured out how to do most stateful workloads on new kubernetes, Ceph is performant and stable, migration progress of production services is here: (ocf auth required to view, unfortunately)

https://docs.google.com/spreadsheets/d/1zQEG2pNiFWxhubetlv5JuB2HlsMmGARzqlTozE4hG58/edit#gid=0

(cc @oliver-ni)