[Feature Request] Provide Migrations API

[tylergets]

Is your feature request related to a problem? Please describe. Prisma itself does not offer programmatic access to its migration library, originally reported here: https://github.com/prisma/prisma/issues/4703

Describe the solution you'd like It would be great if Zenstack provided a way to run migrations programmaticly.

Describe alternatives you've considered Running the prisma command via a shell exec

Additional context The main purpose of this for me is to allow applications to bootstrap itself without needing to run the prisma db push command or ship a preseeded DB

[ymc9]

Hi @tylergets ,

ZenStack doesn't integrate with Prisma migrate in any way today. If it does that I think it should do it in a way that's integrated with other core features (access control etc.).

I'm keeping this issue in the backlog for now to collect more feedback.

[andrictham]

@ymc9

If it does that I think it should do it in a way that's integrated with other core features (access control etc.).

I think ZenStack integrating with Prisma migrate (and extending its capabilities) would make a lot of sense to improve the CRUD generation feature.

Having a good workflow around migrations, especially data migrations, is much more necessary when ZenStack auto-generates client-side CRUD hooks based on the schema. Because now the client-side hooks can get out of sync with Prisma Client on the server and our database schema.

This is in contrast to using Prisma on its own, which only uses the schema to migrate the database.

As they stand now, version skew could cost queries from the auto-generated CRUD hooks to fail, if a client uses an old version of generated hooks tied to an old schema.

This is noticeable even in development when you run prisma migrate or db push to migrate your database, and queries from the client on localhost causes the app to crash or queries to fail because the database query is no longer valid.

The best practice to handle version skew of APIs is to version your APIs, so you can maintain the same shape of request/response for older clients.

How do you do that with the auto-generated RPC-style APIs?

I know that I can still manually create my own API routes using tRPC for instance, and calling the enhance() function.

But for developer productivity, it would be nice if this versioning could be handled for me, since ZenStack knows about my schema and it knows how it changes over time.

Currently data migrations are a missing piece of the Prisma workflow. But since Prisma doesn’t handle CRUD API generation and we’re expected to implement our own, that's perhaps okay. We can build our own workflows around Prisma.

With ZenStack however, let's say we follow the “expand and contract” strategy recommended by Prisma to migrate our data, and we drop the column in the “contract” step, wouldn’t old clients break because the frontend hooks mirror the Prisma API which mirrors the database columns?

The way ZenStack couples our frontend query code to our database schema is great for developer productivity, but is unfortunately less ideal when you need to migrate data. It makes our APIs resistant to change.

[ymc9]

Hey @andrictham , thanks for the wait! I think I don't fully understand the versioning scenario yet.

Here's my confusion. In the "expand and contract" process, when you take the "contract" step and finally remove the old column, there's supposed to be no one using that column anymore, right? In the context of ZenStack, the backend enhanced prisma should have been updated in previous deployments to cut over from reading/writing the old columns to the new ones. The frontend hooks should have been regenerated and redeployed, too.

Is this specifically about the browser windows that haven't been refreshed to load the new frontend bundle yet? Or is it about a general mechanism to run multiple API versions that serve different versions of schemas concurrently? So that you can have the client to keep running different versions of hooks for an extended period of time?

[andrictham]

@ymc9

Is this specifically about the browser windows that haven't been refreshed to load the new frontend bundle yet?

Yes, mostly, that would be my biggest concern for a live production app, as there is no good way to guarantee users will not be using an older version, leading to version skew between client and server, unless we do a version check and force clients to refresh.

This is enough of a problem that Vercel specifically has a feature to handle this.

Vercel’s version skew fix wouldn’t work here, though, because the older backend code would break too since it’s tightly coupled to the database schema.

This could become a non-issue if I do a version check and/or wait long enough for clients to phase out of using older client code.

But that strategy would only work if we forgo any sort of offline capability with optimistic mutations, which Tanstack Query supports.

It would also require us to forgo building PWAs, mobile apps, Chrome extensions, desktop apps, or developer APIs with the auto-generated APIs, since those cannot be easily updated.

Or is it about a general mechanism to run multiple API versions that serve different versions of schemas concurrently? So that you can have the client to keep running different versions of hooks for an extended period of time?

I don’t know if there's an easier way to solve version skew, but if there’s an easier way, I’m all for it.

There seems to be two separate issues here:

• Version skew between client and server (our RPC hooks)
• Version skew between server and database (our Prisma client)

Since ZenStack uses a single schema to generate both, it also has the potential ability to resolve both types of version skew.

And at the same time, the ability to serve multiple backward-compatible API versions and handle deprecated or renamed fields gracefully would also be a win for developer productivity.

There are also migration issues where a schema change would require existing data or older requests to be migrated to the new schema, which Prisma doesn’t handle today, but is nevertheless a necessary part of the developer workflow when dealing with migrations.

Since ZenStack is schema-first and already a DSL for writing custom rules (now largely validation and access control rules), I can imagine it being able to also support data migration rules.

This could help simplify simple schema changes like renaming or combining/splitting fields, or complex tasks like relaxing or imposing new constraints on the data or changing data types.

Off the top of my head, this would be rules like how to:

• Backfill newly-required columns that used to be optional with a default value
• Reroute requests to a column or model that was renamed, using aliases to handle requests coming in that still refer to the column or model by its old name
• Gracefully handle cases where a column in a previous schema had different validation rules or data type, and how to reformat it to be compatible with the new rules/type
• Gracefully handle cases where one or more columns are merged into a single column, by writing rules to transform data, like a computed value (say firstName + lastName)
• Gracefully handle cases where a single column is split into multiple columns, by writing rules to transform data (splitting by substring, for instance)

And for version skew between client and server in RPC code, perhaps older clients automatically send a version which signals what schema version they are querying or mutating, which would trigger the relevant data migration rules to run.

There’s some prior art in a declarative DSL for data migrations in Project Cambria, which could serve as an inspiration.