Datastore operations fails after internet fluctuations

Akash-T2S commented 7 months ago

Before opening, please confirm:

[X] I have searched for duplicate or closed issues and discussions.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React Native

Amplify APIs

DataStore

Amplify Version

v5

Amplify Categories

No response

Backend

None

Environment information

``` # Put output below this line # Put output below this line System: OS: macOS 13.3.1 CPU: (8) arm64 Apple M1 Memory: 47.81 MB / 16.00 GB Shell: 5.9 - /bin/zsh Binaries: Node: 14.21.3 - /opt/homebrew/opt/node@14/bin/node npm: 6.14.18 - /opt/homebrew/opt/node@14/bin/npm Watchman: 2023.08.07.00 - /opt/homebrew/bin/watchman Browsers: Chrome: 119.0.6045.105 Safari: 16.4 npmPackages: @aws-amplify/datastore-storage-adapter: 2.0.43 => 2.0.43 @azure/core-asynciterator-polyfill: 1.0.2 => 1.0.2 @babel/core: 7.12.9 => 7.12.9 (7.23.2) @babel/preset-typescript: 7.22.5 => 7.22.5 (7.23.2) @babel/runtime: 7.12.5 => 7.12.5 (7.23.2) @react-native-async-storage/async-storage: 1.13.2 => 1.13.2 (1.19.4) @react-native-community/art: 1.2.0 => 1.2.0 @react-native-community/datetimepicker: 6.7.0 => 6.7.0 @react-native-community/eslint-config: 2.0.0 => 2.0.0 @react-native-community/masked-view: 0.1.11 => 0.1.11 @react-native-community/netinfo: 9.3.10 => 9.3.10 @react-native-community/progress-view: 1.3.2 => 1.3.2 @react-native-community/push-notification-ios: 1.10.1 => 1.10.1 @react-native-firebase/analytics: 17.5.0 => 17.5.0 @react-native-firebase/app: 17.5.0 => 17.5.0 @react-native-firebase/crashlytics: 17.5.0 => 17.5.0 @react-native-firebase/firestore: 17.5.0 => 17.5.0 @react-native-firebase/messaging: 17.5.0 => 17.5.0 @react-native-google-signin/google-signin: 8.2.1 => 8.2.1 @react-native-picker/picker: 2.4.8 => 2.4.8 @react-navigation/core: 3.7.9 => 3.7.9 @react-navigation/native: 3.8.4 => 3.8.4 @react-navigation/web: 1.0.0-alpha.9 => 1.0.0-alpha.9 @svgr/webpack: 5.5.0 => 5.5.0 @tsconfig/react-native: 2.0.3 => 2.0.3 @types/jest: 29.2.1 => 29.2.1 @types/react: 18.0.0 => 18.0.0 (18.2.37) @types/react-test-renderer: 18.0.0 => 18.0.0 HelloWorld: 0.0.1 ReactWebConfigExample: 1.0.0 amazon-cognito-identity-js: 6.3.1 => 6.3.1 amazon-cognito-identity-js/internals: undefined () appcenter: 4.4.5 => 4.4.5 appcenter-analytics: 4.4.5 => 4.4.5 appcenter-crashes: 4.4.5 => 4.4.5 aws-amplify: 5.3.6 => 5.3.6 aws-amplify-react-native: 7.0.2 => 7.0.2 aws-sdk: 2.1431.0 => 2.1431.0 axios: 1.4.0 => 1.4.0 (0.26.0, 0.21.4) babel-jest: 26.6.3 => 26.6.3 babel-loader: 8.2.2 => 8.2.2 babel-plugin-react-native-web: 0.17.1 => 0.17.1 base-64: 1.0.0 => 1.0.0 (0.1.0) compression-webpack-plugin: 10.0.0 => 10.0.0 core-js: 3.32.0 => 3.32.0 (2.6.12, 1.2.7) country-from-iso2: 1.0.1 => 1.0.1 css-loader: 6.4.0 => 6.4.0 dotenv-webpack: 7.0.3 => 7.0.3 enzyme: 3.11.0 => 3.11.0 enzyme-adapter-react-16: 1.15.5 => 1.15.5 eslint: 7.32.0 => 7.32.0 eslint-config-prettier: 7.0.0 => 7.0.0 (6.15.0) eslint-plugin-jest: 24.1.3 => 24.1.3 (22.4.1) eslint-plugin-prettier: 3.2.0 => 3.2.0 (3.1.2) eslint-plugin-react: 7.21.5 => 7.21.5 eslint-plugin-react-hooks: 4.3.0 => 4.3.0 file-loader: 6.2.0 => 6.2.0 file-saver: 2.0.5 => 2.0.5 firebase: 9.22.0 => 9.22.0 firebase/analytics: undefined () firebase/app: undefined () firebase/app-check: undefined () firebase/auth: undefined () firebase/auth/cordova: undefined () firebase/auth/react-native: undefined () firebase/compat: undefined () firebase/compat/analytics: undefined () firebase/compat/app: undefined () firebase/compat/app-check: undefined () firebase/compat/auth: undefined () firebase/compat/database: undefined () firebase/compat/firestore: undefined () firebase/compat/functions: undefined () firebase/compat/installations: undefined () firebase/compat/messaging: undefined () firebase/compat/performance: undefined () firebase/compat/remote-config: undefined () firebase/compat/storage: undefined () firebase/database: undefined () firebase/firestore: undefined () firebase/firestore/lite: undefined () firebase/functions: undefined () firebase/installations: undefined () firebase/messaging: undefined () firebase/messaging/sw: undefined () firebase/performance: undefined () firebase/remote-config: undefined () firebase/storage: undefined () geo-point-in-polygon: 1.0.0 => 1.0.0 hello-world: 0.0.1 hermes-inspector-msggen: 1.0.0 html-webpack-plugin: 5.3.2 => 5.3.2 husky: 4.3.8 => 4.3.8 ignite-cli: 8.4.2 => 8.4.2 inline-style-prefixer: 6.0.1 => 6.0.1 jest: 26.6.3 => 26.6.3 jest-enzyme: 7.1.2 => 7.1.2 jetifier: 1.6.8 => 1.6.8 libphonenumber-js: 1.10.14 => 1.10.14 libphonenumber-js/build: undefined () libphonenumber-js/core: undefined () libphonenumber-js/max: undefined () libphonenumber-js/max/metadata: undefined () libphonenumber-js/min: undefined () libphonenumber-js/min/metadata: undefined () libphonenumber-js/mobile: undefined () libphonenumber-js/mobile/examples: undefined () libphonenumber-js/mobile/metadata: undefined () lint-staged: 10.5.3 => 10.5.3 localforage: 1.10.0 => 1.10.0 lodash: 4.17.21 => 4.17.21 lottie-ios: 3.4.1 => 3.4.1 (3.1.8) lottie-react-native: 5.1.4 => 5.1.4 (4.1.3) md5: 2.3.0 => 2.3.0 metro-react-native-babel-preset: 0.70.3 => 0.70.3 (0.66.2) mockdate: 3.0.5 => 3.0.5 moment: 2.29.4 => 2.29.4 moment-timezone: 0.5.39 => 0.5.39 msw: 0.35.0 => 0.35.0 pako: 1.0.11 => 1.0.11 (2.0.4) patch-package: 6.5.0 => 6.5.0 performance-now: 2.1.0 => 2.1.0 postman-collection: 4.1.0 => 4.1.0 prettier: 2.8.0 => 2.8.0 promise.allsettled: 1.0.7 => 1.0.7 prop-types: 15.7.2 => 15.7.2 (15.8.1, 15.5.8) qs: 6.11.2 => 6.11.2 (6.11.0, 6.5.3) query-string: 7.0.1 => 7.0.1 (6.14.1, 6.10.1) react: 17.0.2 => 17.0.2 react-datetime: 3.2.0 => 3.2.0 react-devtools: 4.26.1 => 4.26.1 react-dom: 17.0.2 => 17.0.2 react-easy-crop: 4.6.2 => 4.6.2 react-icomoon: 2.5.4 => 2.5.4 react-localization: 1.0.19 => 1.0.19 react-native: 0.67.5 => 0.67.5 react-native-calendars: 1.1291.1 => 1.1291.1 react-native-camera: 4.2.1 => 4.2.1 react-native-circular-progress: 1.3.7 => 1.3.7 react-native-code-push: 7.0.5 => 7.0.5 react-native-config: 1.4.11 => 1.4.11 react-native-contacts: 7.0.5 => 7.0.5 react-native-country-picker-modal: 2.0.0 => 2.0.0 react-native-device-info: 10.3.0 => 10.3.0 react-native-document-picker: 8.1.3 => 8.1.3 react-native-draggable-flatlist: 2.6.2 => 2.6.2 react-native-element-dropdown: 2.4.0 => 2.4.0 react-native-exit-app: 1.1.0 => 1.1.0 react-native-external-display: 0.5.18 => 0.5.18 react-native-fast-image: 8.6.3 => 8.6.3 react-native-fs: 2.20.0 => 2.20.0 react-native-gesture-handler: 1.10.3 => 1.10.3 react-native-get-random-values: 1.9.0 => 1.9.0 react-native-image-crop-picker: 0.38.1 => 0.38.1 react-native-image-picker: 4.10.1 => 4.10.1 react-native-keyboard-aware-scroll-view: 0.9.5 => 0.9.5 react-native-largelist: 3.1.0-rc.2 => 3.1.0-rc.2 react-native-linear-gradient: 2.6.2 => 2.6.2 react-native-loader: 1.3.1 => 1.3.1 react-native-maps: 1.2.0 => 1.2.0 react-native-modal: 13.0.1 => 13.0.1 react-native-modal-datetime-picker: 14.0.0 => 14.0.0 react-native-permissions: 3.6.1 => 3.6.1 (2.2.2) react-native-phone-input: 1.3.5 => 1.3.5 react-native-popup-menu: 0.16.1 => 0.16.1 react-native-print: 0.10.0 => 0.10.0 react-native-progress: 5.0.0 => 5.0.0 react-native-push-notification: 8.1.1 => 8.1.1 react-native-qrcode-scanner: 1.5.5 => 1.5.5 react-native-qrcode-svg: 6.1.2 => undefined (6.1.2, ) react-native-reanimated: 1.13.4 => 1.13.4 react-native-render-html: 6.3.4 => 6.3.4 react-native-restart: 0.0.24 => 0.0.24 react-native-safe-area-context: 3.3.2 => 3.3.2 react-native-screens: 2.16.1 => 2.16.1 react-native-select-contact: 1.6.3 => 1.6.3 react-native-send-intent: 1.3.0 => 1.3.0 react-native-sha1: 1.2.3 => 1.2.3 react-native-simple-download-manager: 1.4.1 => 1.4.1 react-native-sound: 0.11.2 => 0.11.2 react-native-splash-screen: 3.3.0 => 3.3.0 react-native-spring-scrollview: 3.0.1-rc.5 => 3.0.1-rc.5 react-native-sqlite-storage: 6.0.1 => 6.0.1 react-native-svg: 13.6.0 => 13.6.0 react-native-swipe-list-view: 3.2.5 => 3.2.5 react-native-swiper: 1.6.0 => 1.6.0 react-native-switch-selector: 2.2.1 => 2.2.1 react-native-tab-view: 2.15.2 => 2.15.2 react-native-vector-icons: 9.2.0 => 9.2.0 react-native-version-number: 0.3.6 => 0.3.6 react-native-walkthrough-tooltip: 1.4.0 => 1.4.0 react-native-web: 0.17.1 => 0.17.1 react-native-web-linear-gradient: 1.1.2 => 1.1.2 react-native-web-lottie: 1.4.4 => 1.4.4 react-native-web-maps: 0.3.0 => 0.3.0 react-native-web-swiper: 2.2.4 => 2.2.4 react-native-web-webview: 1.0.2 => 1.0.2 react-native-webview: 11.24.0 => 11.24.0 react-native-zohosalesiq-mobilisten: 6.0.0 => 6.0.0 react-navigation: 4.4.4 => 4.4.4 react-navigation-drawer: 2.7.1 => 2.7.1 react-navigation-hooks: 1.1.0 => 1.1.0 react-navigation-stack: 1.8.1 => 1.8.1 react-navigation-tabs: 2.7.0 => 2.7.0 react-redux: 7.2.8 => 7.2.8 react-test-renderer: 18.0.0 => 18.0.0 (16.14.0) reactotron-react-native: 5.0.3 => 5.0.3 reactotron-redux: 3.1.3 => 3.1.3 redux: 4.2.0 => 4.2.0 redux-persist: 6.0.0 => 6.0.0 redux-persist-filesystem-storage: 3.0.0 => 3.0.0 redux-persist/integration/react: undefined () redux-saga: 1.2.2 => 1.2.2 redux-saga-tester: 1.0.874 => 1.0.874 redux-saga/effects: undefined () rn-fetch-blob: 0.12.0 => 0.12.0 rn-placeholder: 3.0.3 => 3.0.3 style-loader: 3.3.0 => 3.3.0 ts-loader: 9.4.4 => 9.4.4 typescript: 4.9.5 => 4.9.5 universal-cookie: 4.0.4 => 4.0.4 url-loader: 4.1.1 => 4.1.1 use-file-picker: 1.5.1 => 1.5.1 uuid: 9.0.0 => 9.0.0 (3.4.0, 8.3.2, 8.0.0, 7.0.3) npmGlobalPackages: corepack: 0.15.1 n: 9.1.0 npm: 6.14.18 ```

Describe the bug

I have been using Amplify Datastore for the past 2 years and after the update to version 5.x.x I've encountered an issue with intermittent internet connectivity. Whenever there are fluctuations in the network, the networkStatus of the HubListener first reports false and then true. After a full or delta sync, any Datastore action fails and throws errors, rendering Datastore unusable until I refresh the app.

The issue is depend on the next action after the internet fluctuations.

Issue: DataStoreStateError: Tried to execute DataStore.query()/DataStore.Start() while DataStore was "Stopping". This can only be done while DataStore is "Started" or "Stopped". To remedy: Ensure all calls to stop() and clear() have completed first. If this is not possible, retry the operation until it succeeds

Once internet is back usual delta sync too fails at times. This can be reproduced in both Android & iOS devices

Expected behavior

Datastore operations should perform even after internet fluctuations and should not affect the usage of it.

Reproduction steps

1) Sync models 2) Once AppSync event is ready 3) Pull out the internet cable if Wifi is connected and put back 4) Repeat step 3 twice or thrice 5) Turn off the Wifi and Turn on back 6) Connect back the internet 7) Wait for delta sync to happen 8) Delta sync fails and other operations will also fail

Code Snippet

// Put your code below this line.
DataStore.configure({
                authProviders: {
                    functionAuthProvider: async () => {
                        return {
                            token: licenseKey
                        };
                    }
                },
                syncExpressions: getSyncExpression(storeId, settingsTimeZone),
                syncPageSize: 1000,
                maxRecordsToSync: 100000,
                fullSyncInterval: FULL_SYNC_INTERVAL,
                storageAdapter: SQLiteAdapter,
                errorHandler: (error) => {

                }
            });

Log output

``` // Put your logs below this line ```

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

david-mcafee commented 7 months ago

Hi @Akash-T2S! I have a few follow-up questions to help us better understand the issue:

Are you upgrading from V4 to V5? If so, have you already taken a look at the migration steps in the README?
It looks like you're using Amplify version 5.3.6 - have you attempted to upgrade to the latest version of V5 (5.3.16)?
Have you attempted to upgrade to the latest version of @aws-amplify/datastore-storage-adapter (2.1.17)?
Are you calling any of the DataStore lifecycle functions (i.e. start, stop, or clear)? If so, could you provide a code snippet demonstrating how, and where, you are calling them?

Thank you!

Akash-T2S commented 7 months ago

@david-mcafee Thanks for these steps,

Step 1 - has been followed thoroughly and implemented in the same way. Step 2 & 3 - I shall update to the mentioned version and get back. Step 4 - DataStore lifecycle functions are triggered as below

DS Configure:

const configureDataStore = () => {
    DataStore.configure({
        authProviders: {
            functionAuthProvider: async () => {
                return {
                    token: licenseKey
                };
            }
        },
        syncExpressions: getSyncExpression(storeId, settingsTimeZone),
        syncPageSize: 1000,
        maxRecordsToSync: 100000,
        fullSyncInterval: FULL_SYNC_INTERVAL,
        storageAdapter: SQLiteAdapter,
        errorHandler: (error) => {}
    });
}

DS Life cycle:

const clearDataStore = async () => {
    await DataStore.clear();
};
const stopDataStore = async () => {
    await DataStore.stop();
};
const startDataStore = async () => {
    await DataStore.start();
};

DS Initialisation:

useEffect(() => {
    configureDataStore();
    startDataStore();
    if (!hubListener) {
        hubListener = Hub.listen('datastore', async (hubData) => {
            if (event === 'ready') {
                const order = await DataStore.query(Order, (c) =>
                  c.and((c) => [
                      c.storeId.eq(storeId),
                      c.deliveryAt.ge(getGraterThanDatesForSyncExpression(settingsTimeZone))
                  ])
                );
            }

        }
    }
    return () => {
        removeOrderSubscription();
        removePrinterObjSubscription();
        if (isValidElement(hubListener)) {
            hubListener();
            hubListener = null;
        }
        if (isValidElement(pubsubListner)) {
            pubsubListner();
            pubsubListner = null;
        }
    };
}, []);

cwomack commented 6 months ago

@Akash-T2S, can you share any relevant code where you're calling DataStore.clear() and DataStore.stop (or based on your code snippets for the DS Lifecycles, clearDataStore and stopDataStore respectively?

Akash-T2S commented 6 months ago

@cwomack clearDataStore will be called only on logout. Whereas stopDataStore will be called when there is any KeepAliveTimeOut happens (KA acknowledgement is missed) through 'api' channel.

Hub.listen('api', async (data) => {
    const { payload } = data;
    if (payload?.event === CONNECTION_STATE_CHANGE) {
        if (payload?.data?.connectionState === 'ConnectedPendingKeepAlive') {
            await DataStore.stop();
            await DataStore.start();
        }
    }
}

david-mcafee commented 5 months ago

@Akash-T2S - it looks like the failure is occurring when you attempt to perform a DataStore operation (e.g. DataStore.query) while DataStore is still "stopping". To avoid this, you can wait for DataStore to be in a "ready" state (see this section of our docs) before attempting to perform an operation.

Additionally, since DataStore works offline, there shouldn't be a need to stop / start DataStore on connection state changes. Can you help us understand your usecase for stopping / starting DataStore on connection state changes?

Lastly, have you attempted to upgrade to the versions listed in this comment?

Akash-T2S commented 5 months ago

@david-mcafee I agree with your point, I make query’s and other operations only when AppSync ready event trigger(after all model synced). Also, I'm trying to stop and start only when the pending keep alive is missed and not when the transition of network status. Even when I don't have that block of code the same issue happens. This reproducible steps are tricky but tbh it occurs more often in the latest version.

Regarding the version update, No luck on that.

david-mcafee commented 1 month ago

@Akash-T2S - thank you for your response! To confirm, are you saying that the issue is still present even when you completely disable all calls to DataStore.stop() in your app? If you could provide us with more detailed reproduction steps and/or context with regards to how and where you are calling DataStore operations and lifecycle methods, that would help us better understand the issue!

Lastly, you mentioned that you had no luck on updating the Amplify version - can you help us understand the issue so we can assist with unblocking you? Thank you!!

aws-amplify / amplify-js