Data migration upon schema.graphql updates

cfbo commented 3 years ago

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?

No response

If applicable, what version of Node.js are you using?

No response

Amplify CLI Version

5.1.0

What operating system are you using?

Windows

Amplify Categories

storage, api

Amplify Commands

Not applicable

Describe the bug

I am trying to understand if Amplify offers a solution for migrating data upon schema changes. We are using Amplify to build an application that will need to go in production. We have added auth, a graphql api, storage, etc. When applying schema changes (to graphql model), we currently loose all user data, but it's fine because it's not in production yet. Are there any solutions in Amplify? Is there any plan to implement them? If not, I think it would make sense to provide a section in the documentation that tries to guide developers, e.g. something like "Data migration best practices on schema updates", under the GraphQL section. I have read aws-amplify/amplify-cli#2689 and at the time it looks like there was no solution yet for this.

I would really appreciate some advise on what are the best practices for this. Even links to some articles would be very useful.

Expected behavior

Amplify to handle data migration on graphql schema updates; or at least some instructions on the documentation.

Reproduction steps

Build an amplify application with a graphql api (among other functionalities) Invite users to the application and see DynamoDB tables associated to the graphql schema being populated with data Evolve the application which requires changing the schema, adding/modifying properties (types, fields, connections, keys, etc.) Upon amplify build (push or CI/CD), the schema is modified and the DynamoDB tables are modified as well, which leads in many cases to data being lost and application logic being broken.

Additional Notes

We are using a react front-end and fully relying on the Datastore to perform all operations. I guess that using the Datastore adds one more layer of complexity for the data migration solution, based on what I read in https://docs.amplify.aws/lib/datastore/how-it-works/q/platform/js#writing-data-from-the-appsync-console

DataStore is designed primarily for developers to not have to focus on the backend and let your application code and workflow create everything. However, there will be some use cases where you will use the AppSync console, a Lambda function, or other out of band processes to write data (such as batch actions or data migrations) and you might send GraphQL operations without the DataStore client.

By the way, the above section is the only place where (so far) I have found "data migration" being mentioned.

cfbo commented 3 years ago

I have continued researching on this topic and I haven't found any solution yet. I have also read all possible related posts in the Amplify Discord channel. Over the last couple of years many people have asked questions about the topic reported above, but there wasn't any answer. The only relevant answer (I assume from the Amplify team) was that one:

Whichever the updated plans are on this topic, it would be great to get an idea, just so that one can work around it.

I am wondering how many organizations are building real production applications with Amplify, or whether most of them are proof of concepts. I don't see how one can build a production-ready application without a solution around data migration. Again, even getting some guidance through documentation or even a blog posts, mentioning possible/recommended approach to tackle this problem would be more than fantastic (given that at the moment I couldn't find literally nothing).

Any help on this topic would be so much appreciated. Many thanks.

justatribute commented 3 years ago

I too have a system that is getting scarily close to production with no obvious solution for performing data migrations following a schema update. This is a massive gap and will be fatal to many projects. Please could the Amplify team provide some insight into how we should proceed - even if it's just some guidance!

cfbo commented 3 years ago

Any feedback on this topic would be highly appreciated!

cfbo commented 3 years ago

Any update?

pedramp20 commented 3 years ago

This is a missing feature. We are also dealing with this problem!

extheory commented 3 years ago

This would be awesome! Please add this

cfbo commented 3 years ago

Looks like the amplify team is looking into some data migration solution right now: https://github.com/aws-amplify/amplify-cli/issues/1407#issuecomment-912725839

cfbo commented 2 years ago

I am trying to implement a custom solution

Does anyone know anything about configuring a temporary frontend REDIRECT? While the build is ongoing, I would like to (temporarily) redirect frontend calls to a maintenance page so that they don't make changes during that time.

cfbo commented 2 years ago

Below the custom approach that I am trying to implement

cfbo commented 2 years ago

@edwardfoyle While waiting for this feature to be processed and implemented by the amplify team, I was wondering it would possible to get some feedback/guidance on the custom solution that I am building. I have shared my notes in the comment above.

My amplify.yml looks like that:

Below some pseudo code to give an idea:

the prebuild script:

the postBuild script:

Some aspects that I am still figuring out

in general, is this a good approach?
is it okay to run node scripts right in the build env or should I use a different approach (e.g. use lambdas?)
is there a way to roll back changes made to the dynamo DB?
datastore conflicts after upgrading (current plan is to force a datastore clear).

rweintraub-cycleon commented 2 years ago

@cfbo did you continue with this approach? do you have any experiences/tips to share?

edwardfoyle commented 2 years ago

Hi everyone, while we haven't yet addressed all of the concerns mentioned in this thread, we are excited to announce a new @mapsTo directive to help with certain scenarios. It is available in the latest version of the CLI (7.6.14) as a developer preview. To try it out, you don't need to do anything except start using the directive in your schema.

This directive can be used to rename a GraphQL type but retain the original table and data. Usage looks like:

type Article @model @mapsTo(name: "Blog") {
  id: ID!
  title: String!
}

Where "Blog" is the original name of the "Article" type that contains data you want to retain. For more details, check out the docs PR here: https://github.com/aws-amplify/docs/pull/3890/files

aws-amplify / amplify-category-api