Example Project / Documentation

[IsaiahByDayah]

Hey @kevlened, I really love that you've solved a problem I (and I imagine many others) have found ourselves with. I was wondering if you had an open-source project to show how this actually looks in practice (for example: how to handle failed migration data rollbacks, formatting longer descriptions in the migration filename, etc).

More than happy to help set up a skeleton project if that would help, but I wanted to see if you had something open already to look at. Thanks

[kevlened]

Hey @IsaiahByDayah, I'm glad you find it useful! Aside from the very basic tests in this repo, I don't have any examples, so if you'd like to set up a project, I'd gladly link to it in the README.

While I'm no longer working on the product that inspired fireway, I can describe our workflow.

• It was a two person team, so we developed against a shared staging db (local Firestore wasn't great at the time). Sharing didn't cause as much trouble as you'd think, because the migrations mostly added new fields, rather than reformatting or deleting them. This ensured that clients with old code still worked.
• For each migration, we used this format: v0.0.0__some-description.js
• We'd use the --dryrun flag until we felt comfortable
• We'd iterate against staging with multiple migrations (0.1.0, 0.1.1, etc), then open a PR. One PR often had a few migrations
• After merging a PR and promoting to prod, the migrations would run against prod
• Failed migrations - We considered backing up the db before each migration, but that'd take too long and be very large. Instead, we ensured our migrations were idempotent. If we had a failed migration, we'd remove the failed entry in the /fireway/ Firestore collection, then rerun the migration. Everything was also roll-forward, so if there was a formatting error in prod, we'd simply write another migration to correct it.

That's what worked for us. Hope that's helpful.

[IsaiahByDayah]

Yes! This is extremely helpful. Thanks for clarifying things a bit more here.

Did you ever run into any race-condition issues? Like running a migration on prod when users may have been adding/removing data? I assume you could run transactions/batches in the migration but you still may miss any newly added documents no? Would you lock down clients somehow on deploys/migrations?

Just more poking around to try and cover all my bases with an approach like this and making sure I at least consider any potential bumps. Thanks again!

[kevlened]

Good question. Let's use an example:

Let's say you write a migration to rename first_name to firstName

• During the migration, some documents will have firstName and others won't.
• If you release client code before migrating, updated clients must handle both scenarios. To simplify, we released after migrating.
• There's always a time when clients are running outdated code (it's cached for web or mobile clients aren't updated).
• Outdated clients won't use the new field yet, so edits will change first_name. There were two ways we tried to handle this: 1) Cloud Firestore triggers - If onUpdate was triggered on a document with first_name, update firstName too. We stopped using this strategy after we moved almost everything to a single collection (this made everything much more flexible). 2) Cloud Functions - We ultimately sent all write (and eventually, non-realtime read) operations through a web api. This allowed us to update/write firstName if first_name was updated and vice versa. We released the api before the migration. This setup also enabled more complex auth strategies. We moved the api to Cloud Run after getting tired of the Cloud Function cold starts.

By handling the scenario of outdated clients, we avoided migration race conditions.

[IsaiahByDayah]

Ahhh, ok. That makes a lot of sense. I've been toying with the idea of doing all writes via a web API for similar reasons so hearing that you guys also found that to be beneficial helps.

Awesome. Thanks again for sharing how you tackled some of these problems. I think it'll def help me with things in my project!

[kevlened]

Happy to help! You’ll also find that an api for mutations allows you to create document ids on the server, instead of allowing users to create any id they want on the client.

Good luck with your project.