Primary Handoff

[benbjohnson]

Currently, replicas will automatically obtain the lease quickly after the primary shuts down. However, this incurs a small window of downtime. We can remove this window by having the primary handoff the session ID to an up-to-date replica and have that replica become the new primary.

[kentcdodds]

I for one would love this!

[benbjohnson]

This is implemented in #303 although the end user UI for it will be done as part of manual promotion (#299).

The idea is that you can run something like:

$ litefs run -promote -if-candidate -- npx prisma migrate deploy

and that'll promote the local node if it is a candidate and run the prisma migration.

[kentcdodds]

I may have misunderstood this feature. So I need to manually switch the primary when deploying?

[benbjohnson]

So I need to manually switch the primary when deploying?

The idea is that you can include that litefs run command above as part of your Dockerfile command and it'll automatically switch candidate nodes to become primary and then run migrations. The promotion is done via handoff of the lease so there shouldn't be any downtime.

Ideally, you'd deploy two nodes with candidate equal to true in a region and when you run a new deployment those will each come up as primary and attempt to run the migration. Obviously, only the first one will actually need to run the migration since the second node will have the migration replicated to it and that migration will be a no-op.

Does that make sense? I'm not sure if that explanation is any clearer. :)

[kentcdodds]

I'm pretty confused 😅 Hard to write it out, so I recorded it: https://www.loom.com/share/d91e16c1cd2849dba96a42f7c03625d0

[benbjohnson]

@kentcdodds Thanks for the loom. That's a such a great tool.

I see your point about running the migrations before starting the server. SQLite is a single writer database so writes will be blocked by the migration regardless of how they're executed. However, it'd be nice to have the server running so it can queue the write requests until the migration is done.

Maybe it makes sense to specify provide multiple exec commands to LiteFS and they can be run in parallel? Something like:

# Automatically promote node when it starts, if it's a candidate.
promote: true

exec:
  - cmd: "npx prisma migrate deploy"
    if-candidate: true

  - cmd: "npm start"

Is that problematic to run the server and migrations in parallel? I'm not sure how Prisma works on the server process exactly. Does it need the schema before it starts up? Maybe it'd be better to run the migrations serially by default as that could be surprising to devs.

Ideally, you would deploy your schema migrations separately from deploying the code changes that depend on them. Or, explained another way, your application should work on the previous schema & the current schema. That's because replicas will likely receive schema updates replicated to them from the primary before new code is deployed on them.

One more idea. If multiple commands are added to the litefs.yml then we could have the LiteFS proxy server just hold onto incoming write requests and wait until the server process has started.

I'm happy to jump on a zoom call too if that'd be easier. There's a lot of nuance in exactly how deployments happen and it's all kinda confusing. :)

[kentcdodds]

I have several thoughts that would be best shared in a video call sometime. I'll let you know when I loop back on this and we can chat about ideas.

[benbjohnson]

Sounds great, thanks @kentcdodds.