DataStore Query Performance Very Slow

sacrampton commented 3 years ago

Before opening, please confirm:

[X] I have searched for duplicate or closed issues and discussions.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React Native

Amplify APIs

DataStore

Amplify Categories

api

Environment information

``` # Put output below this line ```

Describe the bug

We are porting our AppSync/GraphQL based app to DataStore and what we are finding is that queries on the cached database are very slow. We are also finding that pagination of the queries makes no difference to the performance of the queries.

There are many tables with the largest table having about 8,000 records and most other tables having less than 1000 records. There are only 5 tables/models that have data.

Whether we are connected to the network or working offline it makes no difference to the performance.

In issue #6994 it talks of a first time slowing in performance, but we see consistently slow performance.

I have recorded a video of the performance we are seeing - both online and offline - to give you an idea of the problem we are facing.

https://user-images.githubusercontent.com/6362888/120909860-f74b7180-c6bc-11eb-97dc-300f8d19ab4d.mp4

Expected behavior

The database is local so we should be seeing almost instant responses to our queries. The performance of our existing AppSync GraphQL cache gives us almost instant responses so we expect the same or better performance than the existing AppSync GraphQL database.

Reproduction steps

Not sure how to replicate this as the database I have has 48 models and 200 GSI's and takes about 6 hours of incremental (manual) pushes to deploy into a new environment.

Code Snippet

// Put your code below this line.

Log output

``` // Put your logs below this line ```

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

iPhone 7

Mobile Operating System

14.5

Mobile Browser

Not relevant

Mobile Browser Version

No response

Additional information and screenshots

No response

sacrampton commented 3 years ago

Hi @iartemiev - want to push further into the slowness we are seeing for DeltaSync

DataStore creates a separate table in DynamoDB to manage the DeltaSync called "AmplifyDataStore-ENV".

There are no indexes in this table - just the partition key and sort key - where the partition key is table/date and sort key is time/id/version.

Our database is multi-tenanted - and we deal with assets in industrial plants. So I could have hundreds of other users in other plants making massive amounts of changes. But I might not have any users working my plant. The DeltaSync as I see it is going to have to sort through everyone else's changes just to work out there are zero changes that are going to be applicable to me.

When we initially hydrate the cache we do a base query which uses GSI's to get a quick response.

At the moment I'm seeing DeltaSync take about the same amount of time as the full sync (20 minutes). Today I know I was doing a lot of bulk updating of data in a few different plants through our web back end. Not an unusually large workload. But I am concerned from the slowness I'm seeing in our database and what I see in the DeltaSync table for DynamoDB has me worried that this is not scalable for a multi-tenanted environment.

You've been really good at coming up with solutions to get us moving - hopefully someone else has already come up with a solution to make the DeltaSync run in seconds rather than 20+ minutes.

iartemiev commented 3 years ago

@mjaydeep01 try deleting your node_modules and lock file, then adding the following to your package.json and then running npm install:

  "resolutions": {
    "@aws-amplify/datastore": "rn-sqlite",
    "@aws-amplify/core": "rn-sqlite",
    "@aws-amplify/*": "rn-sqlite"
  },

iartemiev commented 3 years ago

@sacrampton, I think this behavior likely warrants a separate GitHub issue, unless this is somehow related to the on-device database on React Native specifically (AsyncStorage or SQLite).

To better understand what's going on, I have some follow up questions:

How many total records are in the delta sync table in DynamoDB at the time that you're seeing the 20 min delta sync time?
How many of those records are being synced down to the app?
Are you using DataStore.configure to change any of the sync-related settings (e.g., syncPageSize, fullSyncInterval, etc.)? If so, which settings are you using?
Are you seeing roughly the same delta sync performance if you test this in a web app?

sacrampton commented 3 years ago

Hi @iartemiev - have created a separate GitHub issue as requested - #8699

mjaydeep01 commented 3 years ago

@iartemiev, thanks for the new suggestion... npm does not seems to using this resolutions at all. With Yarn install, I am getting following errors..

warning Resolution field "rn-sqlite" has an invalid version entry and may be ignored warning Resolution field "rn-sqlite" has an invalid version entry and may be ignored warning Resolution field "@aws-amplify/*" does not end with a valid package name and will be ignored

yarn --version --> 1.22.5 npm --version --> 7.20.5

Can you please suggest how to resolve this issues?

iartemiev commented 3 years ago

@sacrampton, @jeremy-white, @mjaydeep01 - this feature has been officially released as part of aws-amplify@4.2.6! The documentation change will follow shortly, but in the meantime, here's how you can enable it in your app:

(Note: this new storage adapter is only compatible with React Native CLI-generated apps. Expo support will be added in the future)

Install:

$ npm install aws-amplify @aws-amplify/datastore-storage-adapter react-native-sqlite-storage aws-amplify-react-native amazon-cognito-identity-js @react-native-community/netinfo @react-native-async-storage/async-storage
$ npx pod-install

Enable in your app:

import { DataStore } from 'aws-amplify';
import { SQLiteAdapter } from '@aws-amplify/datastore-storage-adapter';

DataStore.configure({
  storageAdapter: SQLiteAdapter
});

I'm closing this issue, as the new storage adapter addresses the performance concerns expressed in the issue. Please create a new issue if you require assistance with this feature.

sacrampton commented 3 years ago

Hi @iartemiev - thank you for going above and beyond to make this happen

prasadk-foodhub commented 2 years ago

npm install aws-amplify @aws-amplify/datastore-storage-adapter react-native-sqlite-storage aws-amplify-react-native amazon-cognito-identity-js @react-native-community/netinfo @react-native-async-storage/async-storage

SqliteAdapter prints empty object.Please confirm sync is not running after doing this

prasadk-foodhub commented 2 years ago

no such table: Setting no such table: ModelMetadata

This are the errors logs.Please advise .I tried below code

import { DataStore } from 'aws-amplify'; import { SQLiteAdapter } from '@aws-amplify/datastore-storage-adapter';

DataStore.configure({ storageAdapter: SQLiteAdapter });

RemyNtshaykolo commented 2 years ago

no such table: Setting no such table: ModelMetadata

This are the errors logs.Please advise .I tried below code

import { DataStore } from 'aws-amplify'; import { SQLiteAdapter } from '@aws-amplify/datastore-storage-adapter';

DataStore.configure({ storageAdapter: SQLiteAdapter });

Hey have you found a solution for this issue?

aws-amplify / amplify-js