aws-amplify / amplify-js

A declarative JavaScript library for application development using cloud services.
https://docs.amplify.aws/lib/q/platform/js
Apache License 2.0
9.41k stars 2.12k forks source link

DataStore generated sync query results in error "InternalFailure" #6707

Closed peterjungx closed 3 years ago

peterjungx commented 4 years ago

DataStore does not work. Spending hours I have identified the following root cause, which renders DataStore unusable due to downstream problems.

Describe the bug I have several tables defined in my schema.graphql. The generated listX queries all work, but there is one syncX query that fails:

query op{
  syncCompanies{
    items {
      id
    }
  }
}

results in

{
  "data": {
    "syncCompanies": null
  },
  "errors": [
    {
      "path": [
        "syncCompanies"
      ],
      "data": null,
      "errorType": "InternalFailure",
      "errorInfo": null,
      "locations": [
        {
          "line": 30,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "An internal error occurred."
    }
  ]
}

All other syncX queries do work (in the AppSync console and from the App). There is no custom configuration from my side, everything is generated by amplify.

Generated schema in AppSync does not look suspicious.

What does error location (line 30 column 3) refer to in the error message?

To Reproduce ?

What is Configured?

const awsmobile = {
    "aws_project_region": "eu-central-1",
    "aws_cognito_identity_pool_id": "eu-central-1:bfa2c3b8-29ab-48bf-b31a-c203c91d12c6",
    "aws_cognito_region": "eu-central-1",
    "aws_user_pools_id": "eu-central-1_9ZshJoqYZ",
    "aws_user_pools_web_client_id": "3c8vqgilt6lrsfgms1llg7g19h",
    "oauth": {},
    "aws_appsync_graphqlEndpoint": "https://efijlzxctre73fh7wyypkp7fra.appsync-api.eu-central-1.amazonaws.com/graphql",
    "aws_appsync_region": "eu-central-1",
    "aws_appsync_authenticationType": "AMAZON_COGNITO_USER_POOLS",
    "aws_cloud_logic_custom": [
        {
            "name": "stripewebhook",
            "endpoint": "https://6psqnj6s9c.execute-api.eu-central-1.amazonaws.com/dev",
            "region": "eu-central-1"
        },
        {
            "name": "webhookapi",
            "endpoint": "https://tz7i5kj70b.execute-api.eu-central-1.amazonaws.com/dev",
            "region": "eu-central-1"
        }
    ],
    "aws_user_files_s3_bucket": "soda-vc149343e68e240dba511955bd41c8b00-dev",
    "aws_user_files_s3_bucket_region": "eu-central-1",
    "aws_mobile_analytics_app_id": "878cd812fb51425994f875846259b61b",
    "aws_mobile_analytics_app_region": "eu-central-1"
};
Environment System: OS: macOS 10.15.6 CPU: (8) x64 Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz Memory: 337.63 MB / 16.00 GB Shell: 5.7.1 - /bin/zsh Binaries: Node: 14.8.0 - /usr/local/bin/node Yarn: 1.22.4 - /usr/local/bin/yarn npm: 6.14.7 - /usr/local/bin/npm Browsers: Chrome: 85.0.4183.83 Firefox: 80.0.1 Safari: 13.1.2 npmPackages: @mdi/font: ^4.4.95 => 4.9.95 @vue/cli-plugin-babel: ^3.4.0 => 3.12.1 @vue/cli-plugin-eslint: ^3.4.0 => 3.12.1 @vue/cli-service: ^3.4.0 => 3.12.1 aws-amplify: ^3.0.24 => 3.0.24 aws-amplify-vue: ^2.1.2 => 2.1.2 babel-eslint: ^10.0.1 => 10.1.0 eslint: ^5.8.0 => 5.16.0 eslint-config-eslint: ^5.0.1 => 5.0.1 eslint-plugin-vue: ^5.2.3 => 5.2.3 eslint-plugin-vuetify: ^1.0.0-beta.3 => 1.0.0-beta.7 file-loader: ^4.1.0 => 4.3.0 graphql-tag: ^2.9.0 => 2.11.0 js-search: ^1.4.2 => 1.4.3 js-yaml-loader: ^1.2.2 => 1.2.2 jsondiffpatch: ^0.4.1 => 0.4.1 markdown-it: ^10.0.0 => 10.0.0 markdown-it-icons: ^0.4.1 => 0.4.1 material-components-web: ^0.35.0 => 0.35.2 moment: ^2.24.0 => 2.27.0 pdfmake: ^0.1.56 => 0.1.68 pdfmake-unicode: ^0.0.1 => 0.0.1 sass: ^1.22.12 => 1.26.10 sass-loader: ^7.0.1 => 7.3.1 stylus: ^0.54.5 => 0.54.8 stylus-loader: ^3.0.1 => 3.0.2 vue: ^2.6.10 => 2.6.12 vue-cli-plugin-vuetify: ^0.4.6 => 0.4.6 vue-event-hub: ^1.0.2 => 1.0.3 vue-mdc-adapter: ^0.15.0 => 0.15.4 vue-moment: ^4.0.0 => 4.1.0 vue-router: ^3.0.2 => 3.4.3 vue-template-compiler: ^2.5.21 => 2.6.12 vuetify: 2.0.16 => 2.0.16 vuetify-loader: ^1.0.5 => 1.6.0 vuex: ^3.1.0 => 3.5.1 xlsx: ^0.14.2 => 0.14.5 npmGlobalPackages: @aws-amplify/cli: 4.27.3 electron: 1.8.8 marked: 1.1.0 npm: 6.14.7 orca: 1.2.1

Additional context We are in the process of integrating DataStore into our product. We started off before DataStore was available and we need a migration plan. (We already know how to port our queries to support the new generated schema directly by taking into account columns _version and _deleted into account, and this approach works.) Raw data in DynamoDB has been touched to pre-fill _version and _lastUpdated columns after enabling DataStore for entire API.

Due to above mentioned process, the DataStore is broken in such a way that syncing into client works for some tables but not for most (for instance await DataStore.query(Userdata) will fetch records, while await DataStore.query(Process) won't.) Syncing from client to backend does not work for any table (e.g. await DataStore.query(new Userdata({'fullname':'ABC'})) will put the entity into local storage but the new item won't be synced). This is probably because the failure of the above query messes up the initialization process of the DataStore client side?

Any help is much appreciated!

peterjungx commented 4 years ago

Here is an excerpt from relevant logs with Amplify.Logger.LOG_LEVEL = 'DEBUG'; enabled.

[DEBUG] 57:31.468 Util -  attempt #1 with this vars: ["query operation($limit: Int, $nextToken: String, $lastSync: AWSTimestamp){\n\t\tsyncUserdata(limit: $limit, nextToken: $nextToken, lastSync: $lastSync){\n\t\t\titems {\n\t\t\t\t\t\t\tid\nfullname\navv_link\navv_version\navv_signature\navv_signed_date\ncreatedAt\nupdatedAt\n_version\n_lastChangedAt\n_deleted\n\t\t\t\t\t\t}\n\t\t\t\t\t\tnextToken\n\t\t\t\t\t\tstartedAt\n\t\t}\n\t}",{"limit":1000,"nextToken":null,"lastSync":1599044155242}] ConsoleLogger.js:97
[DEBUG] 57:31.469 Util -  attempt #1 with this vars: ["query operation($limit: Int, $nextToken: String, $lastSync: AWSTimestamp){\n\t\tsyncCompanies(limit: $limit, nextToken: $nextToken, lastSync: $lastSync){\n\t\t\titems {\n\t\t\t\t\t\t\tid\nname\nfullname\naddress_street\naddress_street_no\naddress_zip\naddress_city\naddress_country\nresponsible_name\nresponsible_role\ncreatedAt\nupdatedAt\narchived_at\ndeleted_at\n_version\n_lastChangedAt\n_deleted\n\t\t\t\t\t\t}\n\t\t\t\t\t\tnextToken\n\t\t\t\t\t\tstartedAt\n\t\t}\n\t}",{"limit":1000,"nextToken":null,"lastSync":0}] ConsoleLogger.js:97
[DEBUG] 57:31.470 Util -  attempt #1 with this vars: ["query operation($limit: Int, $nextToken: String, $lastSync: AWSTimestamp){\n\t\tsyncChangeLogs(limit: $limit, nextToken: $nextToken, lastSync: $lastSync){\n\t\t\titems {\n\t\t\t\t\t\t\tid\nuserId\ncompanyId\nentityType\nentityId\nchangeType\nchangeDoc\ncreatedAt\n_version\n_lastChangedAt\n_deleted\n\t\t\t\t\t\t}\n\t\t\t\t\t\tnextToken\n\t\t\t\t\t\tstartedAt\n\t\t}\n\t}",{"limit":1000,"nextToken":null,"lastSync":1599044155272}] ConsoleLogger.js:97
[DEBUG] 57:31.640 Util - error on  
{…}
​
data: Object { syncCompanies: null }
​
errors: Array [ {…} ]
​
<prototype>: Object { … }
ConsoleLogger.js:107
[DEBUG] 57:31.641 Util -  retrying in 263.69774732341614 ms

and then syncing will be retried 12 times, after that DataStore gives up.

meducati commented 4 years ago

Out of curiosity, are you getting the same errors as me via Utils - error ?? https://github.com/aws-amplify/amplify-js/issues/6691#issue-689000532 (My issue is similar, and also dies after 12 retries)

peterjungx commented 4 years ago

meducati, thanks for asking. No I do not see the timeout errors you reported in #6691. When I open that errors array in the console I always see just one item with "errorType": "InternalFailure" as reported above.

peterjungx commented 4 years ago

A new finding is this: This error does not occur when I bootstrap a new env with amplify add env. Same schema.

So I'd like to avoid the hassle of recreating the production env. This would involve migrating data not only for the API but also cognito. Assuming the problem persists, what would be a suggested better strategy?

Also of interest: Is it possible that the internal state of the AppSync service for my API is somehow messed up? I don't think the data itself is causing this. And if it is, migration could be futile...

Hideman85 commented 4 years ago

I have the same issue just the mentioned line is different in mine https://github.com/aws-amplify/amplify-js/issues/6630#issuecomment-687256401.

In my case this is a fresh app but with a bigger schema (should be scalable so shouldn't be the cause).

PS: Once this error happen DataStore only work locally for the queries & mutations nothing is no longer sync back to the backend.

iartemiev commented 3 years ago

@peterjungx apologies for the delayed response. Are you by chance saving empty arrays/lists as values in your application? E.g.,

DataStore.save(new Company({
  employees: []
});

I've seen the InternalFailure error before in that context.

If not, could you please provide the schema you're using?

rsousa commented 3 years ago

I'm having the problem @iartemiev described, I have an array of strings that can be empty. Datastore stores it and sends it to DynamoDB that also stores it without problems. Later, when datastore tries to do the sync (or if I run the query SyncX in AppSync) I get the "Internal Failure".

Is AWS already targeting this? Is there a work around? I'm thinking to change the type from String[] to String and store a JSON object.

iartemiev commented 3 years ago

@rsousa AppSync is working on fixing this on their end. We'll update this issue and others as soon as that happens. I would suggest saving empty arrays as [null] or [''] until then.

rsousa commented 3 years ago

@iartemiev thank you for your quick reply and for the workaround. Is there a place I can check the progress of the mentioned bug?

iartemiev commented 3 years ago

@rsousa you can follow this GitHub issue. We'll update it once the AppSync bug fix has been deployed.

pragyajha commented 3 years ago

Hi @iartemiev, We are facing similar issue which is blocking us to use it in production. Is there an ETA on when this issue will be fixed?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

rsousa commented 3 years ago

@iartemiev do you have any updates on this issue?

iartemiev commented 3 years ago

@peterjungx, @pragyajha, @rsousa, this issue has been fixed in AppSync. I have tested saving empty lists ([]) as values, and I no longer encounter the error mentioned in the description. I'm closing the issue. If you happen to be still experiencing it, please let us know, and we will re-open it.

LaurentEscalier commented 3 years ago

@iartemiev I am still seeing this error when updating an AWSJSON field (data) in my AppSync API in eu-central-1.

Screen Shot 2021-04-13 at 14 00 57
iartemiev commented 3 years ago

@LaurentEscalier thank you for letting us know. I'll test with a similar schema on my end to confirm.

LaurentEscalier commented 3 years ago

@iartemiev More data to my problem:

I have posted a longer description of my issue in the CLI project: https://github.com/aws-amplify/amplify-cli/issues/7076

LaurentEscalier commented 3 years ago

By the way this seems to also fail when the value is an empty map {}:

Screen Shot 2021-04-13 at 14 25 06
PupoSDC commented 3 years ago

Sharing our current mitigation in case someone else needs a temporary solution:

const awsJson = {
  stringify: (obj: unknown) => {
    const stringified = JSON.stringify(obj);
    return stringified
      .replace(/\[\]/g, '[null]')
      .replace(/\{\}/g, '{"thisKeyShould-beIgnored":null}');
  },
  parse: (stringified: string) => {
    const unhackedString = stringified
      .replace(/\[null\]/g, '[]')
      .replace(/\{"thisKeyShould-beIgnored":null\}/g, '{}');
    return JSON.parse(unhackedString);
  },
};

based of @iartemiev suggestion.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

PupoSDC commented 3 years ago

not stale. we still have to check our way around this.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 3 years ago

This issue has been automatically closed because of inactivity. Please open a new issue if are still encountering problems.

chrisbonifacio commented 3 years ago

Hi :wave: Sorry this was auto-closed by stale bot. We are working towards preventing this from happening in the future. Please let us know if you are still experiencing this issue and in need of assistance.

PupoSDC commented 3 years ago

FWIW we had to abandon amplify for an unrelated reason, so we are no longer "waiting" on this.

Good luck!

chrisbonifacio commented 3 years ago

@peterjungx Closing this issue for now as we have not heard back from you in over a week to confirm if any of the fixes mentioned above have helped to resolve your issue. If you're still experiencing this issue even after the fix by the AppSync team, please comment to let us know and we will re-open.

However, we'd like to continue looking into any related/similar issues such as the one mentioned by @LaurentEscalier regarding the empty objects. If anyone else is still experiencing this behavior please open your own issue and fill out the Bug Report form with any relevant information such as the model definition causing the error to help us with our reproduction of your particular issue.

Thank you!

github-actions[bot] commented 2 years ago

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels or Discussions for those types of questions.