Explore Switching to Kubernetes

[billdybas]

Seems more production-ready than Docker Swarm. May have additional benefits to the Node API.

And I need an excuse to play around w/ it.

[7imbrook]

can I have a few VMs? We can potentially just use Rancher to manage the k8s nodes. Ask about it in 🔒sse-tech.

[Tjzabel]

For a lazy implementation, there is a kubernetes tool to convert docker-compose.yml files over to kubernetes resources.

[Tjzabel]

By the way, I can help with the kubernetes migration if you guys decide to do that. I'll be on campus spring semester if some of you want to meet up to discuss how a migration would work.

[billdybas]

Thinking about this, Kubernetes might be a bit overkill for the Node API & OneRepo. High availability is not necessarily something we need & w/ egg's frequent power outages or random timeouts, unless we had more nodes outside of RIT (which defeats the purpose of using eggs and not outside paid vendors), wouldn't really be worth it. And going the k8s route leads to a rabbit hole of complex, enterprise-buzzword technologies that are likely not a good fit for budding tech heads who are just learning how to write web services.

What we really want is ease of deployment. That's what our Docker setup right now gives us – a declarative way to say what we want running, and an easy way to trigger re-deploys from CI.

I wonder how much more operationally complex this would be, but if we were to run our own PaaS on eggs, it would enable more members to run & deploy SSE apps w/o having to seek their own hosting solutions (eg. like Course Planner's Mongo instance). And we could likely leverage the existing Dockerfiles or write Heroku-style Procfiles w/ buildpacks to make this happen.

Some open-source examples are:

• Flynn: https://flynn.io/
• Dokku: http://dokku.viewdocs.io/dokku/

[Tjzabel]

Overall, the pros and cons (in my opinion) of introducing kubernetes to the SSE are:

Pros

• Bring operation of SSE projects under one "master" location
• Introduce proper failover with mutliple nodes (if available)
• Allow ease of project scalability
• Create namespaces for permissioning
  • Allows for a more clear prod/dev environment, and enables the use of sub-namespaces to segregate projects

Cons

• Would take at least a semester to complete integration
• All involved with the infra would need to be fully ramped up with k8s
• Require overhaul of current infrastructure
• Projects moving underneath k8s would need to be modified slightly to fit new workflow

Achieving these goals would most likely require a large overhaul of how the SSE infrastructure is currently set up. Furthermore, anyone touching the infra would then need to be brought up to date with how the k8s environment runs.

The learning curve for k8s is much higher than that of docker. For simply having ease of deployment, the current docker setup should give you what you need. It's then up to you to decide whether or not adding kubernetes would significantly decrease current issues enough to outweigh all cons.

[akath20]

I have to agree with Bill here, simplicity and ease-of-transferability takes precedence over the benefits k8 offers us.

[Tjzabel]

@akath20 that's the synopsis I got as well. Introducing k8s would add way too few benefits for the amount of trouble it would take to get it all operating.

[billdybas]

I realize there’s probably some context you’re missing from when we migrated to Docker Swarm on eggs.

Twice before this setup we were on eggs and then there was an incident where our network cable was disconnected over the summer so we were unable to access any of our services until people got back in the Fall.

We then moved to Digital Ocean for a few months in the effort to have higher availability. I think this is when we introduced Docker Swarm. We ran into some pushback from the SE department because they’re not big fans of recurring charges on our shared credit card. And we couldn’t really justify paying $20 / month when we have eggs and all it’s compute power available.

So we moved back to eggs and put Swarm on it instead of our DO VPC.

So the current requirements, which may now be more flexible, are:

• Use eggs
• If we use outside services (eg. Mailgun), we must stick to their free tier
• We want low operational complexity since tech heads are not expected to be professional SREs
• We want something that can be deployed from CI easily
• We want our services to be easily replicatable anywhere; eggs shouldn’t be a special snowflake server
• We want our system to restart gracefully after shutdown/power failure

Anything multi-node is probably out of this picture due to the cost constraint. And if we’re using outside servers, why use eggs at all? I’m not sure if ITS would support this either - sharing resources between on- and off-campus servers. And again, we don’t really need HA. And we don’t need scalability either - most projects are webservers with low traffic.

All of our projects are in one location on eggs w/ Swarm. The only thing outside is Course Planner’s Mongo instance. It is a chore to get new projects up and running on the Swarm cluster.

A PaaS seems to meet our goals here since it allows anyone to theoretically deploy with a git push. And we could tie this into CI. Tech’s role could be less about provisioning environments for projects and making website improvements.

My hesitation is in the unknown:

• Do these PaaS do well recovering from power outages?
• How do we allow only certain people to deploy?
• How much time does it take to maintain/upgrade these systems?
• How resource intensive are they and do they perform well on one node?
• If something goes really wrong, how easy is it to debug these systems and get things back online?

[Tjzabel]

Ah, okay. Thanks for the backstory!

I didn't realize you guys only have eggs to work with. Kubernetes is definitely out of the question then. It's not really the best thing to use in a one-node setup.

In terms of using tools like Dokku, I know a few folks over at #rit-foss on IRC (freenode) have a decent amount of experience using Dokku. If you want some of your questions answered in relation to PaaS, I'm sure some can be answered over there.

[Tjzabel]

Also, I'm not actually familiar with what eggs itself is, but it may be worthwhile to have an Ansible playbook of eggs in case something happens to it. This will help fix the "special snowflake" problem.

[billdybas]

Eggs is currently running Ubuntu server (16.04 I think) and has Docker and Docker Swarm running, configured to start up when the server is powered on. It has a few user accounts on it.

[7imbrook]

fly me out there and I'll give a talk on it.