superfly / litefs

FUSE-based file system for replicating SQLite databases across a cluster of machines
Apache License 2.0
3.78k stars 89 forks source link

Allow me to choose a preferred primary without downtime on deploys #170

Closed kentcdodds closed 1 year ago

kentcdodds commented 1 year ago

Based on my website's traffic, my primary should probably be in the United States. However, I don't want to use static leasing because 1) it would be annoying to figure out how to dynamically set that static.primary value based on the region that's deploying and 2) if I understand correctly it means there's a tiny bit of downtime during deploys.

So, what I would prefer is to be able to specify a preferred region that is my primary region and have fly "figure it out" as far as how to choose that region as the primary without any downtime.

Maybe during a deploy, fly starts up the app in the preferred region first and once that one is ready to go, all traffic is diverted over to that region while the others boot up?

I'm way out of my element here, so allow me to just say: As a user, I want to specify a region as the primary without having downtime on deploys.

benbjohnson commented 1 year ago

You can specify which nodes are candidates by setting candidate: true in the config. https://fly.io/docs/litefs/config/#leadership-settings

If you always want your primary to be in the den region, you can could set an environment variable (e.g. IS_PRIMARY_REGION) based on whether FLY_REGION is den and use that in the config. The config is env var aware.

candidate: ${IS_PRIMARY_REGION}

We (Fly.io) don't currently have a concept of "primary" region outside of our Fly Postgres support. I think we're probably going to move in that direction though as we transition our apps from our original Nomad system to our new Fly Machines.

That being said, with a distributed application, it's usually best to deploy database changes so that the old version and new version can both work at the same time since you'll always have overlap if you want to eliminate downtime.

kentcdodds commented 1 year ago

old version and new version can both work at the same time

😅 You're right. That makes loads of sense. Thanks!

benbjohnson commented 1 year ago

@kentcdodds I'm on stream if you have any questions.

kentcdodds commented 1 year ago

How expressive can I be in this YAML file? Can I do this?

candidate: ${FLY_REGION == "den"}
benbjohnson commented 1 year ago

@kentcdodds Right now it only works for $VAR and ${VAR} forms but I like the idea of doing basic equality/inequality checks. It doesn't seem too hard either. I wrote up a separate issue for that: https://github.com/superfly/litefs/issues/176

benbjohnson commented 1 year ago

@kentcdodds I went ahead and just implemented it. It was pretty straightforward and kinda fun to dig into the variable expansion code in Go. :)

Here's the PR: https://github.com/superfly/litefs/pull/177

It's on the latest Docker image (flyio/litefs:sha-653ff14) too.