search

aws-amplify / amplify-js

A declarative JavaScript library for application development using cloud services.
https://docs.amplify.aws/lib/q/platform/js
Apache License 2.0
9.44k stars 2.13k forks source link

Datastore fullsync is very slow #12528

Closed Akash-T2S closed 11 months ago

Akash-T2S commented 1 year ago

Before opening, please confirm:

JavaScript Framework

React Native

Amplify APIs

DataStore

Amplify Categories

storage

Environment information

``` # Put output below this line System: OS: macOS 13.3.1 CPU: (8) arm64 Apple M1 Memory: 47.81 MB / 16.00 GB Shell: 5.9 - /bin/zsh Binaries: Node: 14.21.3 - /opt/homebrew/opt/node@14/bin/node npm: 6.14.18 - /opt/homebrew/opt/node@14/bin/npm Watchman: 2023.08.07.00 - /opt/homebrew/bin/watchman Browsers: Chrome: 119.0.6045.105 Safari: 16.4 npmPackages: @aws-amplify/datastore-storage-adapter: 2.0.43 => 2.0.43 @azure/core-asynciterator-polyfill: 1.0.2 => 1.0.2 @babel/core: 7.12.9 => 7.12.9 (7.23.2) @babel/preset-typescript: 7.22.5 => 7.22.5 (7.23.2) @babel/runtime: 7.12.5 => 7.12.5 (7.23.2) @react-native-async-storage/async-storage: 1.13.2 => 1.13.2 (1.19.4) @react-native-community/art: 1.2.0 => 1.2.0 @react-native-community/datetimepicker: 6.7.0 => 6.7.0 @react-native-community/eslint-config: 2.0.0 => 2.0.0 @react-native-community/masked-view: 0.1.11 => 0.1.11 @react-native-community/netinfo: 9.3.10 => 9.3.10 @react-native-community/progress-view: 1.3.2 => 1.3.2 @react-native-community/push-notification-ios: 1.10.1 => 1.10.1 @react-native-firebase/analytics: 17.5.0 => 17.5.0 @react-native-firebase/app: 17.5.0 => 17.5.0 @react-native-firebase/crashlytics: 17.5.0 => 17.5.0 @react-native-firebase/firestore: 17.5.0 => 17.5.0 @react-native-firebase/messaging: 17.5.0 => 17.5.0 @react-native-google-signin/google-signin: 8.2.1 => 8.2.1 @react-native-picker/picker: 2.4.8 => 2.4.8 @react-navigation/core: 3.7.9 => 3.7.9 @react-navigation/native: 3.8.4 => 3.8.4 @react-navigation/web: 1.0.0-alpha.9 => 1.0.0-alpha.9 @svgr/webpack: 5.5.0 => 5.5.0 @tsconfig/react-native: 2.0.3 => 2.0.3 @types/jest: 29.2.1 => 29.2.1 @types/react: 18.0.0 => 18.0.0 (18.2.37) @types/react-test-renderer: 18.0.0 => 18.0.0 HelloWorld: 0.0.1 ReactWebConfigExample: 1.0.0 amazon-cognito-identity-js: 6.3.1 => 6.3.1 amazon-cognito-identity-js/internals: undefined () appcenter: 4.4.5 => 4.4.5 appcenter-analytics: 4.4.5 => 4.4.5 appcenter-crashes: 4.4.5 => 4.4.5 aws-amplify: 5.3.6 => 5.3.6 aws-amplify-react-native: 7.0.2 => 7.0.2 aws-sdk: 2.1431.0 => 2.1431.0 axios: 1.4.0 => 1.4.0 (0.26.0, 0.21.4) babel-jest: 26.6.3 => 26.6.3 babel-loader: 8.2.2 => 8.2.2 babel-plugin-react-native-web: 0.17.1 => 0.17.1 base-64: 1.0.0 => 1.0.0 (0.1.0) compression-webpack-plugin: 10.0.0 => 10.0.0 core-js: 3.32.0 => 3.32.0 (2.6.12, 1.2.7) country-from-iso2: 1.0.1 => 1.0.1 css-loader: 6.4.0 => 6.4.0 dotenv-webpack: 7.0.3 => 7.0.3 enzyme: 3.11.0 => 3.11.0 enzyme-adapter-react-16: 1.15.5 => 1.15.5 eslint: 7.32.0 => 7.32.0 eslint-config-prettier: 7.0.0 => 7.0.0 (6.15.0) eslint-plugin-jest: 24.1.3 => 24.1.3 (22.4.1) eslint-plugin-prettier: 3.2.0 => 3.2.0 (3.1.2) eslint-plugin-react: 7.21.5 => 7.21.5 eslint-plugin-react-hooks: 4.3.0 => 4.3.0 file-loader: 6.2.0 => 6.2.0 file-saver: 2.0.5 => 2.0.5 firebase: 9.22.0 => 9.22.0 firebase/analytics: undefined () firebase/app: undefined () firebase/app-check: undefined () firebase/auth: undefined () firebase/auth/cordova: undefined () firebase/auth/react-native: undefined () firebase/compat: undefined () firebase/compat/analytics: undefined () firebase/compat/app: undefined () firebase/compat/app-check: undefined () firebase/compat/auth: undefined () firebase/compat/database: undefined () firebase/compat/firestore: undefined () firebase/compat/functions: undefined () firebase/compat/installations: undefined () firebase/compat/messaging: undefined () firebase/compat/performance: undefined () firebase/compat/remote-config: undefined () firebase/compat/storage: undefined () firebase/database: undefined () firebase/firestore: undefined () firebase/firestore/lite: undefined () firebase/functions: undefined () firebase/installations: undefined () firebase/messaging: undefined () firebase/messaging/sw: undefined () firebase/performance: undefined () firebase/remote-config: undefined () firebase/storage: undefined () geo-point-in-polygon: 1.0.0 => 1.0.0 hello-world: 0.0.1 hermes-inspector-msggen: 1.0.0 html-webpack-plugin: 5.3.2 => 5.3.2 husky: 4.3.8 => 4.3.8 ignite-cli: 8.4.2 => 8.4.2 inline-style-prefixer: 6.0.1 => 6.0.1 jest: 26.6.3 => 26.6.3 jest-enzyme: 7.1.2 => 7.1.2 jetifier: 1.6.8 => 1.6.8 libphonenumber-js: 1.10.14 => 1.10.14 libphonenumber-js/build: undefined () libphonenumber-js/core: undefined () libphonenumber-js/max: undefined () libphonenumber-js/max/metadata: undefined () libphonenumber-js/min: undefined () libphonenumber-js/min/metadata: undefined () libphonenumber-js/mobile: undefined () libphonenumber-js/mobile/examples: undefined () libphonenumber-js/mobile/metadata: undefined () lint-staged: 10.5.3 => 10.5.3 localforage: 1.10.0 => 1.10.0 lodash: 4.17.21 => 4.17.21 lottie-ios: 3.4.1 => 3.4.1 (3.1.8) lottie-react-native: 5.1.4 => 5.1.4 (4.1.3) md5: 2.3.0 => 2.3.0 metro-react-native-babel-preset: 0.70.3 => 0.70.3 (0.66.2) mockdate: 3.0.5 => 3.0.5 moment: 2.29.4 => 2.29.4 moment-timezone: 0.5.39 => 0.5.39 msw: 0.35.0 => 0.35.0 pako: 1.0.11 => 1.0.11 (2.0.4) patch-package: 6.5.0 => 6.5.0 performance-now: 2.1.0 => 2.1.0 postman-collection: 4.1.0 => 4.1.0 prettier: 2.8.0 => 2.8.0 promise.allsettled: 1.0.7 => 1.0.7 prop-types: 15.7.2 => 15.7.2 (15.8.1, 15.5.8) qs: 6.11.2 => 6.11.2 (6.11.0, 6.5.3) query-string: 7.0.1 => 7.0.1 (6.14.1, 6.10.1) react: 17.0.2 => 17.0.2 react-datetime: 3.2.0 => 3.2.0 react-devtools: 4.26.1 => 4.26.1 react-dom: 17.0.2 => 17.0.2 react-easy-crop: 4.6.2 => 4.6.2 react-icomoon: 2.5.4 => 2.5.4 react-localization: 1.0.19 => 1.0.19 react-native: 0.67.5 => 0.67.5 react-native-calendars: 1.1291.1 => 1.1291.1 react-native-camera: 4.2.1 => 4.2.1 react-native-circular-progress: 1.3.7 => 1.3.7 react-native-code-push: 7.0.5 => 7.0.5 react-native-config: 1.4.11 => 1.4.11 react-native-contacts: 7.0.5 => 7.0.5 react-native-country-picker-modal: 2.0.0 => 2.0.0 react-native-device-info: 10.3.0 => 10.3.0 react-native-document-picker: 8.1.3 => 8.1.3 react-native-draggable-flatlist: 2.6.2 => 2.6.2 react-native-element-dropdown: 2.4.0 => 2.4.0 react-native-exit-app: 1.1.0 => 1.1.0 react-native-external-display: 0.5.18 => 0.5.18 react-native-fast-image: 8.6.3 => 8.6.3 react-native-fs: 2.20.0 => 2.20.0 react-native-gesture-handler: 1.10.3 => 1.10.3 react-native-get-random-values: 1.9.0 => 1.9.0 react-native-image-crop-picker: 0.38.1 => 0.38.1 react-native-image-picker: 4.10.1 => 4.10.1 react-native-keyboard-aware-scroll-view: 0.9.5 => 0.9.5 react-native-largelist: 3.1.0-rc.2 => 3.1.0-rc.2 react-native-linear-gradient: 2.6.2 => 2.6.2 react-native-loader: 1.3.1 => 1.3.1 react-native-maps: 1.2.0 => 1.2.0 react-native-modal: 13.0.1 => 13.0.1 react-native-modal-datetime-picker: 14.0.0 => 14.0.0 react-native-permissions: 3.6.1 => 3.6.1 (2.2.2) react-native-phone-input: 1.3.5 => 1.3.5 react-native-popup-menu: 0.16.1 => 0.16.1 react-native-print: 0.10.0 => 0.10.0 react-native-progress: 5.0.0 => 5.0.0 react-native-push-notification: 8.1.1 => 8.1.1 react-native-qrcode-scanner: 1.5.5 => 1.5.5 react-native-qrcode-svg: 6.1.2 => undefined (6.1.2, ) react-native-reanimated: 1.13.4 => 1.13.4 react-native-render-html: 6.3.4 => 6.3.4 react-native-restart: 0.0.24 => 0.0.24 react-native-safe-area-context: 3.3.2 => 3.3.2 react-native-screens: 2.16.1 => 2.16.1 react-native-select-contact: 1.6.3 => 1.6.3 react-native-send-intent: 1.3.0 => 1.3.0 react-native-sha1: 1.2.3 => 1.2.3 react-native-simple-download-manager: 1.4.1 => 1.4.1 react-native-sound: 0.11.2 => 0.11.2 react-native-splash-screen: 3.3.0 => 3.3.0 react-native-spring-scrollview: 3.0.1-rc.5 => 3.0.1-rc.5 react-native-sqlite-storage: 6.0.1 => 6.0.1 react-native-svg: 13.6.0 => 13.6.0 react-native-swipe-list-view: 3.2.5 => 3.2.5 react-native-swiper: 1.6.0 => 1.6.0 react-native-switch-selector: 2.2.1 => 2.2.1 react-native-tab-view: 2.15.2 => 2.15.2 react-native-vector-icons: 9.2.0 => 9.2.0 react-native-version-number: 0.3.6 => 0.3.6 react-native-walkthrough-tooltip: 1.4.0 => 1.4.0 react-native-web: 0.17.1 => 0.17.1 react-native-web-linear-gradient: 1.1.2 => 1.1.2 react-native-web-lottie: 1.4.4 => 1.4.4 react-native-web-maps: 0.3.0 => 0.3.0 react-native-web-swiper: 2.2.4 => 2.2.4 react-native-web-webview: 1.0.2 => 1.0.2 react-native-webview: 11.24.0 => 11.24.0 react-native-zohosalesiq-mobilisten: 6.0.0 => 6.0.0 react-navigation: 4.4.4 => 4.4.4 react-navigation-drawer: 2.7.1 => 2.7.1 react-navigation-hooks: 1.1.0 => 1.1.0 react-navigation-stack: 1.8.1 => 1.8.1 react-navigation-tabs: 2.7.0 => 2.7.0 react-redux: 7.2.8 => 7.2.8 react-test-renderer: 18.0.0 => 18.0.0 (16.14.0) reactotron-react-native: 5.0.3 => 5.0.3 reactotron-redux: 3.1.3 => 3.1.3 redux: 4.2.0 => 4.2.0 redux-persist: 6.0.0 => 6.0.0 redux-persist-filesystem-storage: 3.0.0 => 3.0.0 redux-persist/integration/react: undefined () redux-saga: 1.2.2 => 1.2.2 redux-saga-tester: 1.0.874 => 1.0.874 redux-saga/effects: undefined () rn-fetch-blob: 0.12.0 => 0.12.0 rn-placeholder: 3.0.3 => 3.0.3 style-loader: 3.3.0 => 3.3.0 ts-loader: 9.4.4 => 9.4.4 typescript: 4.9.5 => 4.9.5 universal-cookie: 4.0.4 => 4.0.4 url-loader: 4.1.1 => 4.1.1 use-file-picker: 1.5.1 => 1.5.1 uuid: 9.0.0 => 9.0.0 (3.4.0, 8.3.2, 8.0.0, 7.0.3) npmGlobalPackages: corepack: 0.15.1 n: 9.1.0 npm: 6.14.18 ```

Describe the bug

I have 8 to 9 models in my schema with more records in few models. The "modelSynced" event is experiencing a significant delay before it triggers. This delay is caused by the slow insertion of records into SQLite, which varies in duration, ranging from 1 minute to 6 minutes, depending on the data we have for each model. I'm not noticing any delay in the actual data synchronization process. However, there is a noticeable time gap between the last data synchronization and the "modelSynced" & "syncQueriesReady" event, which seems to be substantial.

Expected behavior

The "ModelSynced" events should be triggered immediately upon completion of the synchronization process. Additionally, the synchronized records should be inserted into SQLite without any delay.

Reproduction steps

1) Happens only on Fullsync

Code Snippet

// Put your code below this line.

Log output

``` // Put your logs below this line ```

aws-exports.js

aws_appsync_authenticationType: 'API_LAMBA'

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

chrisbonifacio commented 1 year ago

Hi @Akash-T2S 👋 thanks for raising this issue.

Since you are building a React Native app, can you confirm which platform you're experiencing this behavior in?

Android? iOS? Or both?

Akash-T2S commented 1 year ago

@chrisbonifacio we have this issue in almost all the Android devices.

chrisbonifacio commented 1 year ago

There is a known issue/limitation with the Android implementation of DataStore where records are written to the SQLite database one at a time, resulting in the delay between sync and ready events. This issue does not exist in iOS because records are all written in one transaction.

chrisbonifacio commented 11 months ago

Hi 👋 Closing this as we have not heard back from you. If you are still experiencing this issue and in need of assistance, please feel free to comment and provide us with any information previously requested by our team members so we can re-open this issue and be better able to assist you.

Thank you!

Akash-T2S commented 11 months ago

@chrisbonifacio Do we have a plan to fix this anytime?

Akash-T2S commented 1 month ago

@chrisbonifacio @nadetastic Any alternative we can do for this? As we have a fatal issue due to this behaviour.