search

aws-amplify / amplify-js

A declarative JavaScript library for application development using cloud services.
https://docs.amplify.aws/lib/q/platform/js
Apache License 2.0
9.44k stars 2.13k forks source link

Selective Sync IndexedDB Adapter Nondeterministic Results #8810

Closed sherl0cks closed 2 years ago

sherl0cks commented 3 years ago

Before opening, please confirm:

JavaScript Framework

Not applicable

Amplify APIs

DataStore

Amplify Categories

api

Environment information

NPM package for React App ``` npx envinfo --system --binaries --browsers --npmPackages --duplicates --npmGlobalPackages System: OS: macOS 11.5.2 CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz Memory: 1.27 GB / 16.00 GB Shell: 5.8 - /bin/zsh Binaries: Node: 14.17.5 - ~/.asdf/installs/nodejs/14.17.5/bin/node npm: 6.14.14 - ~/.asdf/plugins/nodejs/shims/npm Browsers: Chrome: 92.0.4515.159 Firefox: 89.0.2 Safari: 14.1.2 npmPackages: @apollo/client: ^3.3.21 => 3.3.21 @apollo/client/cache: undefined () @apollo/client/core: undefined () @apollo/client/errors: undefined () @apollo/client/link/batch: undefined () @apollo/client/link/batch-http: undefined () @apollo/client/link/context: undefined () @apollo/client/link/core: undefined () @apollo/client/link/error: undefined () @apollo/client/link/http: undefined () @apollo/client/link/persisted-queries: undefined () @apollo/client/link/retry: undefined () @apollo/client/link/schema: undefined () @apollo/client/link/utils: undefined () @apollo/client/link/ws: undefined () @apollo/client/react: undefined () @apollo/client/react/components: undefined () @apollo/client/react/context: undefined () @apollo/client/react/data: undefined () @apollo/client/react/hoc: undefined () @apollo/client/react/hooks: undefined () @apollo/client/react/parser: undefined () @apollo/client/react/ssr: undefined () @apollo/client/testing: undefined () @apollo/client/utilities: undefined () @babel/plugin-proposal-nullish-coalescing-operator: ^7.14.2 => 7.14.2 (7.12.1) @material-ui/core: ^4.11.4 => 4.11.4 @testing-library/jest-dom: ^5.12.0 => 5.12.0 @testing-library/react: ^11.2.6 => 11.2.6 @testing-library/user-event: ^12.8.3 => 12.8.3 @turf/distance: ^6.3.0 => 6.3.0 @types/earcut: ^2.1.1 => 2.1.1 @types/geojson: ^7946.0.7 => 7946.0.7 @types/jest: ^26.0.23 => 26.0.23 @types/leaflet: ^1.7.0 => 1.7.4 @types/lodash: ^4.14.168 => 4.14.169 @types/mathjs: ^6.0.12 => 6.0.12 @types/node: ^12.20.12 => 12.20.12 @types/randomcolor: ^0.5.5 => 0.5.5 @types/react: ^17.0.5 => 17.0.5 @types/react-dom: ^17.0.3 => 17.0.3 @types/react-router-dom: ^5.1.8 => 5.1.8 @types/three: ^0.127.1 => 0.127.1 earcut: ^2.2.2 => 2.2.2 exponential-backoff: ^3.1.0 => 3.1.0 leaflet: ^1.7.1 => 1.7.1 leaflet-path-drag: ^1.1.0 => 1.1.0 lodash: ^4.17.21 => 4.17.21 mathjs: ^9.3.2 => 9.3.2 randomcolor: ^0.6.2 => 0.6.2 react: ^17.0.2 => 17.0.2 react-app-rewired: ^2.1.8 => 2.1.8 react-dom: ^17.0.2 => 17.0.2 react-jss: ^10.7.1 => 10.7.1 react-leaflet: ^3.2.0 => 3.2.0 react-router-dom: ^5.2.0 => 5.2.0 react-scripts: 4.0.3 => 4.0.3 three: ^0.128.0 => 0.128.0 typescript: ^4.2.4 => 4.2.4 web-vitals: ^1.1.1 => 1.1.1 ``` Separate DataStore npm package ``` npx envinfo --system --binaries --browsers --npmPackages --duplicates --npmGlobalPackages System: OS: macOS 11.5.2 CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz Memory: 1.27 GB / 16.00 GB Shell: 5.8 - /bin/zsh Binaries: Node: 14.17.5 - ~/.asdf/installs/nodejs/14.17.5/bin/node npm: 6.14.14 - ~/.asdf/plugins/nodejs/shims/npm Browsers: Chrome: 92.0.4515.159 Firefox: 89.0.2 Safari: 14.1.2 npmPackages: @apollo/client: ^3.3.21 => 3.3.21 @apollo/client/cache: undefined () @apollo/client/core: undefined () @apollo/client/errors: undefined () @apollo/client/link/batch: undefined () @apollo/client/link/batch-http: undefined () @apollo/client/link/context: undefined () @apollo/client/link/core: undefined () @apollo/client/link/error: undefined () @apollo/client/link/http: undefined () @apollo/client/link/persisted-queries: undefined () @apollo/client/link/retry: undefined () @apollo/client/link/schema: @apollo/client/link/schema: @apollo/client/link/schema: System: OS: macOS 11.5.2 CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz Memory: 1.19 GB / 16.00 GB Shell: 5.8 - /bin/zsh Binaries: Node: 14.17.5 - ~/.asdf/installs/nodejs/14.17.5/bin/node npm: 6.14.14 - ~/.asdf/plugins/nodejs/shims/npm Browsers: Chrome: 92.0.4515.159 Firefox: 89.0.2 Safari: 14.1.2 npmPackages: @sunrun/aws-automation: ^1.0.21 => 1.0.21 @types/geojson: ^7946.0.7 => 7946.0.7 @types/isomorphic-fetch: 0.0.35 => 0.0.35 @types/jest: ^26.0.23 => 26.0.23 @types/lodash: ^4.14.170 => 4.14.170 @types/node: ^15.6.1 => 15.6.1 @types/uuid: ^8.3.0 => 8.3.0 aws-amplify: ^4.2.4 => 4.2.4 aws-appsync: ^4.0.3 => 4.0.3 geojson: ^0.5.0 => 0.5.0 graphql-tag: ^2.11.0 => 2.12.4 isomorphic-fetch: ^3.0.0 => 3.0.0 jest: ^27.0.3 => 27.0.3 (26.6.0) localforage: ^1.10.0 => 1.10.0 lodash: ^4.17.21 => 4.17.21 react-scripts: ^4.0.3 => 4.0.3 redux-persist: ^4.10.2 => 4.10.2 redux-persist/constants: undefined () redux-persist/storages: undefined () ts-jest: ^27.0.2 => 27.0.2 typescript: ^4.3.2 => 4.3.2 uuid: ^8.3.2 => 8.3.2 (3.4.0, 3.3.2) ```

Describe the bug

As context, I am building an application with DataStore and modeling data with the DynamoDB version control pattern.

I have application code in a backend process that is saving Post{ id: 1, version: 0} and Post{ id:1, version:1 }. This process also saves a DataStore record like `Event{ type: 'foo', postId: 1}, which I subscribe to via DataStore.observe. Upon receipt, I update DataStore selective sync according to the docs. In the network tab of the browser, I can see that both Post{ id: 1, version: 0} and Post{ id:1, version:1 } come over the wire on the sync query. However, when inspecting the DataStore indexeddb table for Post, I can see that it reliably only has 1 of the Post records, and that it does not seem deterministic which one is there.

Expected behavior

Both Post{ id: 1, version: 0} and Post{ id:1, version:1 } are stored in IndexedDB and available via DataStore.query.

Reproduction steps

  1. Deploy an Amplify API using the schema provided.
  2. Deploy DataStore code that uses selective sync as described.
  3. Write the Post via the DataStore generated createPost GraphQL mutation. This should be done from a different process than the one querying Posts, whether that is cli / aws console / a different browser - it doesn't matter.
  4. Update selective sync at runtime to query for the new Post Id as described in the code snippet.
  5. Inspect IndexedDB and network tab as described.

Code Snippet

I have the model that follows a structure like: 

type Post @model {
  id: ID!
  version: Int!
}

type Comment @model
  @key(name: "byPost", fields: ["postID", "postVersion"]) {
  id: ID!
  postID: ID!
  postVersion: Int!
}

I also have DataStore selective sync expressions that follow a structure like:

const postId = 'someId'
DataStore.configure({
  syncExpressions: [
    syncExpression(Post, post => post.id('eq', postId)),
    syncExpression(Comment, comment => comment.postID('eq', postId))
  ]
});

Log output

``` // Put your logs below this line ```

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

sherl0cks commented 3 years ago

@jamesaucode this is a really scary bug for me. What can I do to help move forward triage? Do you need a full reproducer?

iartemiev commented 2 years ago

Hi @sherl0cks, are you still experiencing this behavior in the latest version of the library?

sherl0cks commented 2 years ago

@iartemiev we are no longer using datastore because it does not support the dynamodb version control pattern. We are still using amplify, but we interact appsync through a simple axios client.

chrisbonifacio commented 2 years ago

it does not seem deterministic which one is there.

I'm not exactly sure what you mean, the record that is synced and persisted to the client should always be the latest version from the server.

Both Post{ id: 1, version: 0} and Post{ id:1, version:1 } are stored in IndexedDB and available via DataStore.query.

Could you explain why you would expect two versions of the same record to be stored locally instead of the most recent version that was synced from the server? Curious about your use case and how or why DataStore doesn't seem to fulfill that requirement.

sherl0cks commented 2 years ago

@chrisbonifacio we are using the dynamo db version control pattern which requires items to have a primary key with both partition key AND sort key. I’ve been told my aws support that datastore only supports uniquely identifying records by partition key. The version in question here is not the _version from appsync conflict detection, but the version from the dynamo db version control pattern. Items have both fields, one for history, one for conflict detection. This allows users to view the history of the item in question, and thus why we need multiple records.

This works great without datastore. It would nice if datastore supported the use case.

chrisbonifacio commented 2 years ago

@sherl0cks from your explanation of the requirements for your use case (partition and sort keys), it sounds like this might be supported by DataStore in the near future once support for custom primary keys is released. The PR for it has been merged, just waiting on public release.

In the meantime, a non-DataStore enabled GraphQL API would be the way to leverage the dynamodb version control pattern.

Closing this out as an answered question