search

getkuby / kuby-core

A convention over configuration approach for deploying Rails apps. https://getkuby.io
MIT License
580 stars 26 forks source link

Support recent versions of kubedb #54

Closed camertron closed 2 years ago

camertron commented 3 years ago

Unfortunately the fine people over at AppsCode (the makers of kubedb) now require each user to obtain a free license. While kuby could do some tricky things to obtain a license automatically, that doesn't feel particularly moral (or maybe even legal?) The only solution I can think of is to point the user to the AppsCode website during cluster setup and then prompt them for the contents of the license file. Obviously that's not ideal, but I think it's the only option we have if we want to keep using kubedb with kuby (which I think we do - it's awesome).

camertron commented 3 years ago

Ok, some additional context here. It turns out the hard part isn't programmatically fetching a license (which is possible, we just have to prompt people for a license server token, email address, etc). The hard part is upgrading the kubedb operator and individual databases. Uninstalling v0.12 is hard by itself because you obviously don't want to take down any existing database instances, but I eventually got it working. Fortunately installing v0.18 (the latest version at the time of this writing) is pretty easy with Helm... but damn it is super hard to upgrade an existing Postgres or MySQL instance that's already running (with zero downtime). Here are the steps I've (almost) got working for Postgres:

  1. Ensure the existing db is set up for logical replication, which requires a change in WAL level (write-ahead log). Setting the WAL level requires a pg restart, so the easiest way to do this is to set the number of replicas to > 1 while simultaneously telling kubedb to use additional config, which will result in a restart. You want multiple replicas anyway so restarts don't result in downtime.
  2. Uninstall the old version of the kubedb operator and install the new one, taking special care not to bring down your existing database instance(s). This can be a little tricky since k8s is fairly reluctant to delete the Postgres CRD if there are still Postgres instances running. Makes sense. Fortunately there are ways to trick k8s into doing what you want.
  3. Stand up a new postgres instance with WAL = logical and create the necessary pub/sub between the old and the new so any records in the old db are copied to the new db. Be careful to give the postgres user a password in the new db if the user you use to access the db from rails isn't postgres (Kuby uses root by default). This is necessary because of what I believe is a bug in kubedb's init scripts. This new db must obviously have a different name than the old one, which is a problem. The name of the db determines the DNS name rails connects to, which we can't really change. Hmm...
  4. Deploy the rails app and somehow point it at the new database service.
  5. Delete the old database
  6. Profit

I'm currently stuck trying to figure out how to easily point the rails app at the new database service. The service URL is essentially hard-coded and can't change dynamically. Moreover the new database service can't have the same name as the existing one, so we'd have to store the "right" one somewhere (maybe an annotation on the deployment?) and consult it on every... docker build cuz that's when we rewrite the database.yml? UGH.

Another solution could be to use some sort of proxy that could dynamically point traffic at one service or another, like a load balancer but on top of services.

kingdonb commented 2 years ago

While kuby could do some tricky things to obtain a license automatically, that doesn't feel particularly moral (or maybe even legal?)

Tamal is a good fella and I don't think he will mind at all, as long as you expose the details of what is happening to users so they are not surprised when the temporary license expires at the end of the month. (? what happens when the license expires, I wonder now? If it means backups are stopped, that could be bad; if it means other things start failing, also could be bad... if it means you just cannot create new databases for now, until the license is renewed, this might not be so bad.)

There's an option for external database already, so users have the option of continuing with KubeDB or taking databases into their own hands another way, which is frankly not a difficult problem today unless you are not using a cloud provider that has support for the type of database that you need.

Thanks for the walk-through about what is needed to upgrade! I have a license for my KubeDB, which is generously granted to Team Hephy for our production cluster, and I have older databases which are in need of upgrades, that were never started or hosted from KubeDB; I got the license to learn to use the database backups, I have both PostgresQL and MariaDBs, tl;dr I've got my work cut out for me in this area, as well.

I'm just now learning how to operate a proper storage CSI driver with my home lab cluster, and I expect this is going to make a huge difference in how I handle databases and improve the disaster recovery properties of my "production" cluster. 🎉

Kuby could also, in the long run, turn out to be a great sales vector for KubeDB. I don't know how you'll feel about that!

camertron commented 2 years ago

Tamal is a good fella and I don't think he will mind at all, as long as you expose the details of what is happening to users so they are not surprised when the temporary license expires at the end of the month.

Oh the license expires too?? Yikes. I actually reached out to the AppsCode team asking if there was any way to avoid the licensing issues, and they basically said "no" 😏 Oh well, I understand why they've chosen to go that route.

There's an option for external database already, so users have the option of continuing with KubeDB or taking databases into their own hands another way, which is frankly not a difficult problem today unless you are not using a cloud provider that has support for the type of database that you need.

Right, that's what most people are doing with Kuby today because they (rightly) distrust running something as sensitive as their database in k8s.

I'm just now learning how to operate a proper storage CSI driver with my home lab cluster, and I expect this is going to make a huge difference in how I handle databases and improve the disaster recovery properties of my "production" cluster. 🎉

I have to admit to knowing next to nothing about the CSI stuff. If you end up writing about this, I'd be curious to read it :)

Kuby could also, in the long run, turn out to be a great sales vector for KubeDB. I don't know how you'll feel about that!

I just landed a long-running branch that removes KubeDB in favor of CockroachDB, which I think is going to be a much better long-term bet for Kuby than either MySQL or Postgres. Nothing against KubeDB - I still think it's an awesome product. It's just that Cockroach was designed from the beginning to work in cloud environments like k8s.