aws-amplify / amplify-js

A declarative JavaScript library for application development using cloud services.
https://docs.amplify.aws/lib/q/platform/js
Apache License 2.0
9.44k stars 2.13k forks source link

Datastore is not getting subscription updates until you interact with the model #7588

Closed DevTGhosh closed 3 years ago

DevTGhosh commented 3 years ago

Describe the bug When updates to a datastore item is made the updates do not show up in another instance via subscription automatically. They only show up if you interact with the model in some way.

To Reproduce Create a model and and have 2 instances running side by side and update one item the changes are not going to be reflected in the second instance until he uses one of the crud action only then the updates take place.

Expected behavior DataStore should keep the local store in sync with DynamoDB via AppSync subscriptions and they should fire and update items automatically.

Screenshots See the video here. Each row is it's own item in dynamodb. So user A updates items row 1 & 2 it doesn't show up in userB's screen no matter how long i wait. It only shows up once user B updates row 3.

https://user-images.githubusercontent.com/25743418/105140088-3b2e4780-5b1d-11eb-95eb-77049c392d7b.mp4

Degub logs

User A updates debug log, ids and urls are changed

[DEBUG] 31:20.396 DataStore - params ready

[DEBUG] 31:20.443 Util -  attempt #1 with this vars: ["QuoteInfo","Update","{\"id\":\"t89FXbxywVaaMs\",\"quoteId\":\"IuOeDNoc-H86_\",\"tenantId\":\"49b-43e0-428f-b0fe-ea59c3cc\",\"freelancers\":null,\"activeFlag\":true,\"agentCommission\":null,\"agentCommissionPercentage\":null,\"allowMarginAndOverheadPercentageGlobalUpdate\":true,\"language\":\"en\",\"category\":\"Talent\",\"categoryId\":3,\"categoryPrefix\":\"IIA\",\"category_fr\":\"Interprétation\",\"costLineLabel\":\"118A\",\"costLineType\":\"costLine\",\"editable\":true,\"exchangeEnabled\":false,\"exchangeRate\":1,\"tableRowIndex\":23,\"isInternational\":false,\"isVisible\":true,\"item\":\"Casting director\",\"item_fr\":\"Directeur de casting\",\"margin\":null,\"marginPercentage\":null,\"number\":null,\"overhead\":null,\"overheadPercentage\":null,\"overtimeHours\":null,\"overtimeHoursCost\":null,\"payrollTaxEnabled\":\"Yes\",\"payrollTaxType\":\"Shooting crew\",\"quantity\":10,\"rate\":null,\"rateInEuros\":null,\"subCategory\":\"Casting\",\"subCategoryId\":4,\"subCategoryPrefix\":\"3A\",\"subCategorySuffix\":\"A\",\"subCategory_fr\":\"Casting\",\"total\":null,\"totalHeading\":null,\"totalMargin\":null,\"unit\":\"Day\",\"smg\":null,\"overtimeDetails\":[{\"multiplier\":1,\"overtimeHours\":null},{\"multiplier\":2,\"overtimeHours\":null},{\"multiplier\":3,\"overtimeHours\":null},{\"multiplier\":4,\"overtimeHours\":null}],\"_version\":1,\"_lastChangedAt\":1610824752498,\"_deleted\":null}","{}",null,null,{"data":"{\"id\":\"89FXbxywVaaMsXZJ\",\"quoteId\":\"IuOeDNoc-H86lDU_\",\"tenantId\":\"349b-43e0-428f-b0fe-ea59c3cc\",\"freelancers\":null,\"activeFlag\":true,\"agentCommission\":null,\"agentCommissionPercentage\":null,\"allowMarginAndOverheadPercentageGlobalUpdate\":true,\"language\":\"en\",\"category\":\"Talent\",\"categoryId\":3,\"categoryPrefix\":\"IIA\",\"category_fr\":\"Interprétation\",\"costLineLabel\":\"118A\",\"costLineType\":\"costLine\",\"editable\":true,\"exchangeEnabled\":false,\"exchangeRate\":1,\"tableRowIndex\":23,\"isInternational\":false,\"isVisible\":true,\"item\":\"Casting director\",\"item_fr\":\"Directeur de casting\",\"margin\":null,\"marginPercentage\":null,\"number\":null,\"overhead\":null,\"overheadPercentage\":null,\"overtimeHours\":null,\"overtimeHoursCost\":null,\"payrollTaxEnabled\":\"Yes\",\"payrollTaxType\":\"Shooting crew\",\"quantity\":10,\"rate\":null,\"rateInEuros\":null,\"subCategory\":\"Casting\",\"subCategoryId\":4,\"subCategoryPrefix\":\"3A\",\"subCategorySuffix\":\"A\",\"subCategory_fr\":\"Casting\",\"total\":null,\"totalHeading\":null,\"totalMargin\":null,\"unit\":\"Day\",\"smg\":null,\"overtimeDetails\":[{\"multiplier\":1,\"overtimeHours\":null},{\"multiplier\":2,\"overtimeHours\":null},{\"multiplier\":3,\"overtimeHours\":null},{\"multiplier\":4,\"overtimeHours\":null}],\"_version\":1,\"_lastChangedAt\":1610824752498,\"_deleted\":null}","modelId":"9TIPt89FXbxywVaaMsXZJ","model":"QuoteInfo","operation":"Update","condition":"{}","id":"01EWFY402XK8PR5BKACKQ6"}]

[DEBUG] 31:20.444 AuthClass - Getting current session

[DEBUG] 31:20.445 AuthClass - Getting the session from this user: 

[DEBUG] 31:20.446 AuthClass - Succeed to get the user session

[DEBUG] 31:20.446 RestClient - POST https://b6pjastgf5cnxfj4yxsiow43fy.appsync-api.eu-west-2.amazonaws.com/graphql

[DEBUG] 31:21.198 DataStore - params ready

[DEBUG] 31:22.9 AWSAppSyncRealTimeProvider - subscription message from AWS AppSync RealTime: {"id":"939d3caa-8888-4ff2-ac13-480e263","type":"data","payload":{"data":{"onUpdateQuoteInfo":{"id":"9TIPt89FXbxywVaa","quoteId":"IuOeDNoc-H","tenantId":"49b-43e0-428f-b0fe-ea59c3c","freelancers":null,"activeFlag":true,"agentCommission":null,"agentCommissionPercentage":null,"allowMarginAndOverheadPercentageGlobalUpdate":true,"language":"en","category":"Talent","categoryId":3,"categoryPrefix":"IIA","category_fr":"Interprétation","costLineLabel":"118A","costLineType":"costLine","editable":true,"exchangeEnabled":false,"exchangeRate":1.0,"tableRowIndex":23,"isInternational":false,"isVisible":true,"item":"Casting director","item_fr":"Directeur de casting","margin":null,"marginPercentage":null,"number":null,"overhead":null,"overheadPercentage":null,"overtimeHours":null,"overtimeHoursCost":null,"payrollTaxEnabled":"Yes","payrollTaxType":"Shooting crew","quantity":10.0,"rate":null,"rateInEuros":null,"subCategory":"Casting","subCategoryId":4,"subCategoryPrefix":"3A","subCategorySuffix":"A","subCategory_fr":"Casting","total":null,"totalHeading":null,"totalMargin":null,"unit":"Day","smg":null,"overtimeDetails":[{"multiplier":1.0,"overtimeHours":null},{"multiplier":2.0,"overtimeHours":null},{"multiplier":3.0,"overtimeHours":null},{"multiplier":4.0,"overtimeHours":null}],"_version":2,"_lastChangedAt":1611147681490,"_deleted":null}}}}

[DEBUG] 31:22.10 AWSAppSyncRealTimeProvider {id: "caa-8888-4ff2-ac13-480e263", observer: SubscriptionObserver, query: "subscription operation {↵  onUpdateQuoteInfo {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", variables: {…}}

[DEBUG] 31:43.950 AWSAppSyncRealTimeProvider - subscription message from AWS AppSync RealTime: {"type":"ka"}

[DEBUG] 31:43.968 AWSAppSyncRealTimeProvider {id: "", observer: null, query: "", variables: {…}}

User B debug log during the time user A updated one of the items

[DEBUG] 31:21.557 AWSAppSyncRealTimeProvider - subscription message from AWS AppSync RealTime: {"id":"1-e316-48cd-94c5-72246b72","type":"data","payload":{"data":{"onUpdateQuoteInfo":{"id":"bxywVaaMsXZJ","quoteId":"DNoc-H86lDU_","tenantId":"49b-43e0-428f-b0fe-ea59c3cc","freelancers":null,"activeFlag":true,"agentCommission":null,"agentCommissionPercentage":null,"allowMarginAndOverheadPercentageGlobalUpdate":true,"language":"en","category":"Talent","categoryId":3,"categoryPrefix":"IIA","category_fr":"Interprétation","costLineLabel":"118A","costLineType":"costLine","editable":true,"exchangeEnabled":false,"exchangeRate":1.0,"tableRowIndex":23,"isInternational":false,"isVisible":true,"item":"Casting director","item_fr":"Directeur de casting","margin":null,"marginPercentage":null,"number":null,"overhead":null,"overheadPercentage":null,"overtimeHours":null,"overtimeHoursCost":null,"payrollTaxEnabled":"Yes","payrollTaxType":"Shooting crew","quantity":10.0,"rate":null,"rateInEuros":null,"subCategory":"Casting","subCategoryId":4,"subCategoryPrefix":"3A","subCategorySuffix":"A","subCategory_fr":"Casting","total":null,"totalHeading":null,"totalMargin":null,"unit":"Day","smg":null,"overtimeDetails":[{"multiplier":1.0,"overtimeHours":null},{"multiplier":2.0,"overtimeHours":null},{"multiplier":3.0,"overtimeHours":null},{"multiplier":4.0,"overtimeHours":null}],"_version":2,"_lastChangedAt":1611147681490,"_deleted":null}}}}

[DEBUG] 31:22.10 AWSAppSyncRealTimeProvider {id: "caa-8888-4ff2-ac13-480e2638d", observer: SubscriptionObserver, query: "subscription operation {↵  onUpdateQuoteInfo {↵   …  _version↵    _lastChangedAt↵    _deleted↵  }↵}↵", variables: {…}}

[DEBUG] 31:43.950 AWSAppSyncRealTimeProvider - subscription message from AWS AppSync RealTime: {"type":"ka"}

[DEBUG] 31:43.968 AWSAppSyncRealTimeProvider {id: "", observer: null, query: "", variables: {…}}

No network request was made in User B's instance when user A updated.

amhinson commented 3 years ago

Could you share your schema.graphql?

DevTGhosh commented 3 years ago

type QuoteInfo
  @model
  @key(name: "byQuote", fields: ["quoteId"])
  @auth(
    rules: [
      { allow: groups, groups: ["companyAdmin"], operations: [create, read, update, delete] }
      { allow: groups, groups: ["freelancer"], operations: [create, read, update, delete] }
    ]
  ) {
  id: ID!
  quoteId: ID!
  tenantId: ID!
  freelancers: [ID]
  activeFlag: Boolean!
  agentCommission: Float
  agentCommissionPercentage: Float
  allowMarginAndOverheadPercentageGlobalUpdate: Boolean
  language: String
  category: String
  categoryId: Int
  categoryPrefix: String
  category_fr: String
  costLineLabel: String
  costLineType: String
  editable: Boolean
  exchangeEnabled: Boolean
  exchangeRate: Float
  tableRowIndex: Int
  isInternational: Boolean
  isVisible: Boolean
  item: String
  item_fr: String
  margin: Float
  marginPercentage: Float
  number: Float
  overhead: Float
  overheadPercentage: Float
  overtimeDetails: [overtimeDetail]
  overtimeHours: Float
  overtimeHoursCost: Float
  payrollTaxEnabled: String
  payrollTaxType: String
  quantity: Float
  rate: Float
  rateInEuros: Float
  subCategory: String
  subCategoryId: Int
  subCategoryPrefix: String
  subCategorySuffix: String
  subCategory_fr: String
  total: Float
  totalHeading: Float
  totalMargin: Float
  unit: String
  smg: Float
}

type overtimeDetail {
  multiplier: Float
  overtimeHours: Float
}

Update code

      const originalRow = await DataStore.query(QuoteInfo, row.id)
      const parsedOriginalRow = JSON.parse(JSON.stringify(originalRow))
      // uses lodash isEqual(https://lodash.com/docs/#isEqual) to only update if the row being updated is different from row fetched from the datastore
      if (!isEqual(parsedOriginalRow, row)) {
        // updates the og quote and saves it
        const updatedRow = await DataStore.save(
          QuoteInfo.copyOf(originalRow, (updated) => {
            for (const property in newRow) {
              // Only updates the property that is different
              if (!isEqual(row[property], updated[property])) {
                updated[property] = newRow[property]
              }
            }
          })
        )
      }

I am using optimistic consistency as the merge conflict resolution.

amhinson commented 3 years ago

Hmm interesting! I'm trying to reproduce the issue with your schema, but the subscriptions seem to be working properly for me. I've got a few more questions:

DevTGhosh commented 3 years ago

It's the same same user in both instances. I am also facing an issue with the datastore version not getting updated automatically when you update the model item. Because of optimistic merge conflict it first sends a network request with the old version which gets rejected then the update is tried again. I am not sure if that can be the cause of the problem.

Screenshots Update 1 api call msedge_INilwpHONU

Update 2 api call rejected due to version mismatch msedge_sKLlMLMMrV

Update 2 api call automatic retry msedge_F6hEaaPWAE

My package.json

{
  "name": "chedar-v2",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@aws-amplify/api": "^3.2.17",
    "@aws-amplify/auth": "^3.4.13",
    "@aws-amplify/core": "^3.8.5",
    "@aws-amplify/datastore": "^2.8.1",
    "@material-ui/core": "^4.11.1",
    "@material-ui/icons": "^4.9.1",
    "@material-ui/lab": "^4.0.0-alpha.57",
    "@reduxjs/toolkit": "^1.5.0",
    "@testing-library/jest-dom": "^5.11.6",
    "@testing-library/react": "^11.2.2",
    "@testing-library/user-event": "^12.6.0",
    "dayjs": "^1.10.3",
    "fontsource-roboto": "^3.0.3",
    "history": "^5.0.0",
    "i18next": "^19.8.4",
    "lodash": "^4.17.20",
    "react": "^17.0.1",
    "react-data-grid": "7.0.0-alpha.29",
    "react-data-grid-addons": "^7.0.0-alpha.24",
    "react-dom": "^17.0.1",
    "react-hook-form": "^6.12.1",
    "react-i18next": "^11.8.4",
    "react-infinite-scroll-component": "^5.1.0",
    "react-redux": "^7.2.2",
    "react-router": "^6.0.0-beta.0",
    "react-router-dom": "^6.0.0-beta.0",
    "react-scripts": "4.0.1",
    "react-virtualized": "^9.22.2",
    "redux": "^4.0.5",
    "web-vitals": "^0.2.4",
    "workbox-background-sync": "^5.1.4",
    "workbox-broadcast-update": "^5.1.4",
    "workbox-cacheable-response": "^5.1.4",
    "workbox-core": "^5.1.4",
    "workbox-expiration": "^5.1.4",
    "workbox-google-analytics": "^5.1.4",
    "workbox-navigation-preload": "^5.1.4",
    "workbox-precaching": "^5.1.4",
    "workbox-range-requests": "^5.1.4",
    "workbox-routing": "^5.1.4",
    "workbox-strategies": "^5.1.4",
    "workbox-streams": "^5.1.4"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  },
  "devDependencies": {
    "@hookform/devtools": "^2.2.1"
  }
}
amhinson commented 3 years ago

Ah yes, that version issue is likely related to the main issue you are seeing. Could you also try updating to using the latest version of aws-amplify package in your package.json instead of the modular packages? Using them individually can lead to problems due to mismatched package versions and duplicate dependencies. Also since you are using react-scripts, you will benefit from tree-shaking when just using aws-amplify.

amhinson commented 3 years ago

Also, from looking at the screenshots of the mutations, when are updates 1 and 2 happening? From the code snippet you provided, I only see one update.

DevTGhosh commented 3 years ago

Changing it to aws-amplify package completely broke Datastore. Getting the following error when I try to start datastore after logging in. It includes the debug log

[DEBUG] 32:55.418 AuthClass 
[DEBUG] 37:08.147 Credentials - removing aws-amplify-federatedInfo from storage
[DEBUG] 37:08.148 Credentials - set credentials from session
[DEBUG] 37:09.69 Credentials - Load credentials successfully
[DEBUG] 37:09.69 AuthClass - succeed to get cognito credentials
[DEBUG] 37:09.408 Hub - Dispatching to auth with 
[DEBUG] 37:09.409 DataStore - Starting DataStore
[DEBUG] 37:09.410 DataStore - Starting Storage

// Error 

IndexedDBAdapter.ts:114 Uncaught (in promise) TypeError: Cannot read property 'namespaces' of undefined
    at IndexedDBAdapter.<anonymous> (IndexedDBAdapter.ts:114)
    at step (InMemoryStore.ts:41)
    at Object.next (InMemoryStore.ts:41)
    at InMemoryStore.ts:41
    at new Promise (<anonymous>)
    at push../node_modules/aws-amplify/node_modules/@aws-amplify/datastore/lib-esm/storage/adapter/IndexedDBAdapter.js.__awaiter (InMemoryStore.ts:41)
    at upgrade (IndexedDBAdapter.ts:112)
    at IDBOpenDBRequest.<anonymous> (index.js:16

My login code

import { Auth, DataStore, syncExpression } from 'aws-amplify'
  const user = await Auth.signIn(email, password)
  await DataStore.start()

My package.json

"dependencies": {
    "@aws-amplify/datastore": "^2.9.3",
    "@material-ui/core": "^4.11.1",
    "@material-ui/icons": "^4.9.1",
    "@material-ui/lab": "^4.0.0-alpha.57",
    "@reduxjs/toolkit": "^1.5.0",
    "@testing-library/jest-dom": "^5.11.6",
    "@testing-library/react": "^11.2.2",
    "@testing-library/user-event": "^12.6.0",
    "aws-amplify": "^3.3.14",
    }
amhinson commented 3 years ago

Would you be able to provide a larger code sample of all the DataStore configuration and usage, or possibly even a small example repo that shows the behavior you're seeing? I've just had no luck replicating the issues that you are seeing so far.

Also, as an aside, you shouldn't need to include @aws-amplify/datastore in your package.json as well, since it is already included in aws-amplify. This likely isn't causing any issues, but I just wanted to call that out.

iartemiev commented 3 years ago

I agree with what Alex said above, but just want to stress that it's actually common to experience issues with syncing if you have aws-amplify as well as @aws-amplify/datastore in your dependencies. I would start with uninstalling @aws-amplify/datastore and then do a clean install of your dependencies with rm -rf node_modules && yarn for good measure.

You should only have aws-amplify installed explicitly.

DevTGhosh commented 3 years ago

The rejection issue was being caused because the version number was manually being updated.

The subscription issue was because I was using redux to display data. I still can't see any websocket data being passed but it wouldn't have mattered for my use case since redux can't handle websocket connections by default.

raybotha commented 3 years ago

Changing it to aws-amplify package completely broke Datastore. Getting the following error when I try to start datastore after logging in. It includes the debug log

[DEBUG] 32:55.418 AuthClass 
[DEBUG] 37:08.147 Credentials - removing aws-amplify-federatedInfo from storage
[DEBUG] 37:08.148 Credentials - set credentials from session
[DEBUG] 37:09.69 Credentials - Load credentials successfully
[DEBUG] 37:09.69 AuthClass - succeed to get cognito credentials
[DEBUG] 37:09.408 Hub - Dispatching to auth with 
[DEBUG] 37:09.409 DataStore - Starting DataStore
[DEBUG] 37:09.410 DataStore - Starting Storage

// Error 

IndexedDBAdapter.ts:114 Uncaught (in promise) TypeError: Cannot read property 'namespaces' of undefined
    at IndexedDBAdapter.<anonymous> (IndexedDBAdapter.ts:114)
    at step (InMemoryStore.ts:41)
    at Object.next (InMemoryStore.ts:41)
    at InMemoryStore.ts:41
    at new Promise (<anonymous>)
    at push../node_modules/aws-amplify/node_modules/@aws-amplify/datastore/lib-esm/storage/adapter/IndexedDBAdapter.js.__awaiter (InMemoryStore.ts:41)
    at upgrade (IndexedDBAdapter.ts:112)
    at IDBOpenDBRequest.<anonymous> (index.js:16

My login code

import { Auth, DataStore, syncExpression } from 'aws-amplify'
  const user = await Auth.signIn(email, password)
  await DataStore.start()

My package.json

"dependencies": {
    "@aws-amplify/datastore": "^2.9.3",
    "@material-ui/core": "^4.11.1",
    "@material-ui/icons": "^4.9.1",
    "@material-ui/lab": "^4.0.0-alpha.57",
    "@reduxjs/toolkit": "^1.5.0",
    "@testing-library/jest-dom": "^5.11.6",
    "@testing-library/react": "^11.2.2",
    "@testing-library/user-event": "^12.6.0",
    "aws-amplify": "^3.3.14",
    }

Did you figure out why you got the TypeError: Cannot read property 'namespaces' of undefined error? I'm experiencing the same error randomly and inconsistently after login with the aws-amplify package, and I only have aws-amplify@4.0.0 as a dependency.

dobeerman commented 3 years ago

Still experiencing the issue. Any updates, guys?

IndexedDBAdapter.js?05f2:159 Uncaught (in promise) TypeError: Cannot read property 'namespaces' of undefined
    at IndexedDBAdapter.eval (IndexedDBAdapter.js?05f2:159)
    at step (IndexedDBAdapter.js?05f2:32)
    at Object.eval [as next] (IndexedDBAdapter.js?05f2:13)
    at eval (IndexedDBAdapter.js?05f2:7)
    at new Promise (<anonymous>)
    at __awaiter (IndexedDBAdapter.js?05f2:3)
    at upgrade (IndexedDBAdapter.js?05f2:151)
    at IDBOpenDBRequest.eval (index.js?3f4f:16)

I can't realize the reason of different errors. Some time the error as above during init, some time the error as following when I try to save a new model. The DB is absolutely new and empty.

One of the specified object stores was not found.
DevTGhosh commented 3 years ago

@raybotha & @dobeerman If you are using both the aws-amplify & @aws-amplify/datastore packages remove either one of them and check. There is a conflict usually if both those packages are present. Also make sure to update those packages and your amplify cli to the latest version.

github-actions[bot] commented 2 years ago

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels or Discussions for those types of questions.