Retrieve data from Amplify Cloud back to DataStore

gauravdangi commented 3 years ago

I am new to Amplify datastore and decided to use Amplify Datastore with flutter app to provide offline mode feature to the users. I successfully implemented my project with Datastore but I would like to clear the local storage when the user logs out. And if logs back, I want user's data to be there but it seems like datastore sync data from local to cloud out of the box but do not sync cloud to local storage back since I can't see any records (After login back or after reinstalling).

So are we required to retrieve the data from dynamoDB using GraphQL APIs and then save it to the local datastore or is there any out-of-the-box function provided by Amplify to retrieve data from the cloud? What's the ideal approach?

Logs

I/amplify:aws-datastore(22312): Orchestrator lock acquired.
I/amplify:aws-datastore(22312): Orchestrator lock released.
E/amplify:aws-datastore(22312): Failure encountered while attempting to start API sync.
E/amplify:aws-datastore(22312): DataStoreException{message=Timed out waiting for subscription processor to start., cause=null, recoverySuggestion=Retry}
E/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.SubscriptionProcessor.startSubscriptions(SubscriptionProcessor.java:154)
E/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:309)
E/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$to6jsrpqq5TGk-8kfWyEK_7WfwQ.subscribe(Unknown Source:2)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletableSubscribeOn$SubscribeOnObserver.run(CompletableSubscribeOn.java:64)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
E/amplify:aws-datastore(22312):         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E/amplify:aws-datastore(22312):         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
E/amplify:aws-datastore(22312):         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
E/amplify:aws-datastore(22312):         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
E/amplify:aws-datastore(22312):         at java.lang.Thread.run(Thread.java:923)
W/amplify:aws-datastore(22312): API sync failed - transitioning to LOCAL_ONLY.
W/amplify:aws-datastore(22312): DataStoreException{message=Timed out waiting for subscription processor to start., cause=null, recoverySuggestion=Retry}
W/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.SubscriptionProcessor.startSubscriptions(SubscriptionProcessor.java:154)
W/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:309)
W/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$to6jsrpqq5TGk-8kfWyEK_7WfwQ.subscribe(Unknown Source:2)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletableSubscribeOn$SubscribeOnObserver.run(CompletableSubscribeOn.java:64)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
W/amplify:aws-datastore(22312):         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W/amplify:aws-datastore(22312):         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
W/amplify:aws-datastore(22312):         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/amplify:aws-datastore(22312):         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/amplify:aws-datastore(22312):         at java.lang.Thread.run(Thread.java:923)
I/amplify:aws-datastore(22312): Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLY
I/amplify:aws-datastore(22312): Setting currentState to LOCAL_ONLY
I/amplify:aws-datastore(22312): Stopping subscription processor.
I/amplify:aws-api(22312): No more active subscriptions. Closing web socket.
I/amplify:aws-datastore(22312): Stopped subscription processor.

I also notice after a few logins and logouts it somehow fetched the data from the cloud and I was able to see the records on my application. Not sure what's going on :/

To Reproduce Steps to reproduce the behavior:

Login and generate some data
Clear datastore and logout
uninstall the app and run the application again

Expected behavior After signing and querying Datastore, the datastore should sync with the cloud or provide Datastore functions to retrieve data from the cloud and the user should see the data he/she has created before.

Platform iOS. Didn't try in Android.

Output of flutter doctor -v

```Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel stable, 2.0.3, on macOS 11.2.3 20D91 darwin-arm, locale en-IN) [✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3) [✓] Xcode - develop for iOS and macOS [✓] Chrome - develop for the web [✓] Android Studio (version 4.1) [✓] VS Code (version 1.54.3) [✓] Connected device (1 available) • No issues found!```

Smartphone (please complete the following information):

Device: iPhone 12 Pro Max
OS: iOS 14.5

haverchuck commented 3 years ago

@gauravdangi Did you try using query when the user logs back in?

gauravdangi commented 3 years ago

Yes. And it returned null. This is the query I run when a user logs in:-

await Amplify.DataStore.query(
        User.classType,
        where: User.ID.eq(_userId),
      );

Also want to inform I did await Amplify.DataStore.clear() while logging out the user.

haverchuck commented 3 years ago

@gauravdangi Did you set up your API with the Amplify CLI, and if so, did you choose the "Enable DataStore" option?

Also when you say 'null' do you mean an empty list?

gauravdangi commented 3 years ago

@haverchuck Yes. I have enabled Datastore and now I am getting the "Disable Datastore for entire API" option.

And yes, by null I mean empty list, sorry for the confusion.

Also Just want to confirm, are we required to implement the logic to fetch records via Graphql APIs from the backend, or Datastore query will do the work?

gauravdangi commented 3 years ago

Also, I missed the logs before. I have edited the question with the logs below:

I/amplify:aws-datastore(22312): Orchestrator lock acquired.
I/amplify:aws-datastore(22312): Orchestrator lock released.
E/amplify:aws-datastore(22312): Failure encountered while attempting to start API sync.
E/amplify:aws-datastore(22312): DataStoreException{message=Timed out waiting for subscription processor to start., cause=null, recoverySuggestion=Retry}
E/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.SubscriptionProcessor.startSubscriptions(SubscriptionProcessor.java:154)
E/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:309)
E/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$to6jsrpqq5TGk-8kfWyEK_7WfwQ.subscribe(Unknown Source:2)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletableSubscribeOn$SubscribeOnObserver.run(CompletableSubscribeOn.java:64)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
E/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
E/amplify:aws-datastore(22312):         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E/amplify:aws-datastore(22312):         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
E/amplify:aws-datastore(22312):         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
E/amplify:aws-datastore(22312):         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
E/amplify:aws-datastore(22312):         at java.lang.Thread.run(Thread.java:923)
W/amplify:aws-datastore(22312): API sync failed - transitioning to LOCAL_ONLY.
W/amplify:aws-datastore(22312): DataStoreException{message=Timed out waiting for subscription processor to start., cause=null, recoverySuggestion=Retry}
W/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.SubscriptionProcessor.startSubscriptions(SubscriptionProcessor.java:154)
W/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:309)
W/amplify:aws-datastore(22312):         at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$to6jsrpqq5TGk-8kfWyEK_7WfwQ.subscribe(Unknown Source:2)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.operators.completable.CompletableSubscribeOn$SubscribeOnObserver.run(CompletableSubscribeOn.java:64)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
W/amplify:aws-datastore(22312):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
W/amplify:aws-datastore(22312):         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W/amplify:aws-datastore(22312):         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
W/amplify:aws-datastore(22312):         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/amplify:aws-datastore(22312):         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/amplify:aws-datastore(22312):         at java.lang.Thread.run(Thread.java:923)
I/amplify:aws-datastore(22312): Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLY
I/amplify:aws-datastore(22312): Setting currentState to LOCAL_ONLY
I/amplify:aws-datastore(22312): Stopping subscription processor.
I/amplify:aws-api(22312): No more active subscriptions. Closing web socket.
I/amplify:aws-datastore(22312): Stopped subscription processor.

haverchuck commented 3 years ago

@gauravdangi Did you run amplify push after amplify update? It's correct that you are getting the 'Disable Datastore' option since you've run the update command locally, but you need to publish that change to the cloud.

And yes- DataStore.query should fetch records from the backend, provided that you have added both the DataStore plugin (which I think you have) and the API plugin.

gauravdangi commented 3 years ago

@haverchuck Yes, I have. Below is the output of Amplify status

Current Environment: dev

| Category | Resource name   | Operation | Provider plugin   |
| -------- | --------------- | --------- | ----------------- |
| Auth     | cognitoa948c9c0 | No Change | awscloudformation |
| Api      | mytales         | No Change | awscloudformation |
| Storage  | s30992fa97      | No Change | awscloudformation |

haverchuck commented 3 years ago

@gauravdangi Are you working with an existing data source with a significant number of records?

gauravdangi commented 3 years ago

@haverchuck No. It does not have a significant number of records. Less than 200 records in total.

gauravdangi commented 3 years ago

@haverchuck Also, I am running my application on two different devices (Android and ios) and when I changed the data on my android device, it is not syncing with the one running on iOS. Though I can see that the records were updated in DynamoDB tables. Similarly, the changes made on iOS are not syncing with the one running on android. Is this an expected behavior? Is there a way to sync data programmatically (DynamoDb to DataStore)? It seems like in my app, the Datastore is fetching results from the cached data all the time instead of the cloud.

Below is the example of how I am querying most of my records:

Fetching user's info:

      final userResponse = await Amplify.DataStore.query(
        User.classType,
        where: User.ID.eq(_userId),
      );

In the below query, I had to write two different queries basis on the platform. Created the issue regarding this too

      if (Platform.isAndroid) {
        return await Amplify.DataStore.query(
          Chapter.classType,
          where: Book.ID.eq(id),
          sortBy: [Chapter.DATE.descending()],
        );
      } else {
        return await Amplify.DataStore.query(
          Chapter.classType,
          where: Chapter.BOOK.eq(id),
          sortBy: [Chapter.DATE.descending()],
        );
      }

pubspec.yaml

  amplify_flutter: '<1.0.0'
  amplify_auth_cognito: '<1.0.0'
  amplify_analytics_pinpoint: '<1.0.0'
  amplify_datastore: '<1.0.0'
  amplify_api: '<1.0.0'
  amplify_storage_s3: '<1.0.0'

GraphQL schema:

type User @model
  @auth(
    rules: [
      { allow: owner, ownerField: "owner", operations: [create, update, delete, read] },
      { allow: groups, groups: ["Admin"]}
  ]) {
  id: ID!
  name: String!
  email: String!
  premiumTill: AWSDateTime!
  books: [Book] @connection(keyName: "byUser", fields: ["id"])
}

type Book @model
  @key(name: "byUser", fields: ["userID", "year"]) 
  @auth(
    rules: [
      { allow: owner, ownerField: "owner", operations: [create, update, delete, read] },
            { allow: groups, groups: ["Admin"]}
  ]) {
  id: ID!
  title: String!
  year: Int
  userID: ID!
  user: User @connection(fields: ["userID"])
  chapters: [Chapter] @connection(keyName: "byBook", fields: ["id"])
}

I'm also seeing the below logs:

D/AWSMobileClient(29221): getCredentials: Validated user is signed-in
I/amplify:aws-datastore(29221): Orchestrator lock acquired.
I/amplify:aws-datastore(29221): Orchestrator transitioning from LOCAL_ONLY to SYNC_VIA_API
I/amplify:aws-datastore(29221): Setting currentState to SYNC_VIA_API
I/amplify:aws-datastore(29221): Starting API synchronization mode.
I/amplify:aws-datastore(29221): Orchestrator lock released.
I/amplify:aws-datastore(29221): Orchestrator lock acquired.
I/amplify:aws-datastore(29221): Orchestrator lock released.
W/amplify:aws-api(29221): Thread interrupted waiting for connection acknowledgement
W/amplify:aws-datastore(29221): API sync failed - transitioning to LOCAL_ONLY.
W/amplify:aws-datastore(29221): DataStoreException{message=Error during subscription., cause=ApiException{message=Thread interrupted waiting for connection acknowledgement, cause=null, recoverySuggestion=Sorry, we don't have a suggested fix for this error yet.}, recoverySuggestion=Evaluate details.}
W/amplify:aws-datastore(29221):         at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$subscription$3(AppSyncClient.java:298)
W/amplify:aws-datastore(29221):         at com.amplifyframework.datastore.appsync.-$$Lambda$AppSyncClient$8gXTT9bgLZGP51blif6hkh7EwIc.accept(Unknown Source:4)
W/amplify:aws-datastore(29221):         at com.amplifyframework.api.aws.SubscriptionOperation.lambda$null$1$SubscriptionOperation(SubscriptionOperation.java:92)
W/amplify:aws-datastore(29221):         at com.amplifyframework.api.aws.-$$Lambda$SubscriptionOperation$EfR-sAOj2xzA_-vAoxWBdb2iFk8.accept(Unknown Source:4)
W/amplify:aws-datastore(29221):         at com.amplifyframework.api.aws.SubscriptionEndpoint.requestSubscription(SubscriptionEndpoint.java:127)
W/amplify:aws-datastore(29221):         at com.amplifyframework.api.aws.SubscriptionOperation.lambda$start$2$SubscriptionOperation(SubscriptionOperation.java:83)
W/amplify:aws-datastore(29221):         at com.amplifyframework.api.aws.-$$Lambda$SubscriptionOperation$Uz-v98-sFPcWrrCLK25EF8LKbXo.run(Unknown Source:2)
W/amplify:aws-datastore(29221):         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)
W/amplify:aws-datastore(29221):         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W/amplify:aws-datastore(29221):         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/amplify:aws-datastore(29221):         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/amplify:aws-datastore(29221):         at java.lang.Thread.run(Thread.java:923)
W/amplify:aws-datastore(29221): Caused by: ApiException{message=Thread interrupted waiting for connection acknowledgement, cause=null, recoverySuggestion=Sorry, we don't have a suggested fix for this error yet.}
W/amplify:aws-datastore(29221):         at com.amplifyframework.api.aws.SubscriptionEndpoint.requestSubscription(SubscriptionEndpoint.java:128)
W/amplify:aws-datastore(29221):         ... 7 more
I/amplify:aws-datastore(29221): Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLY
I/amplify:aws-datastore(29221): Setting currentState to LOCAL_ONLY
D/AutoSessionTracker(29221): Activity paused: MainActivity
D/AutoSessionTracker(29221): Activity stopped: MainActivity
D/AutoSessionTracker(29221): Activity state saved: MainActivity

Amplifiyer commented 3 years ago

@gauravdangi, first to answer your question, No, you don't need to sync data from cloud to local manually. When datastore is initialized, it will start the sync process which will sync all the cloud data to local device (e.g. after calling clear as you did)

The problem seems to be from the timeout exception you are encountering as seen in your logs. cause=ApiException{message=Thread interrupted waiting for connection acknowledgement. All the logs are from android does it mean that your iOS device is syncing the data correctly from the cloud (DDB)?

What is your datastore API flow. E.g. when do you call datastore APIs with respect to user signin and signout.

gauravdangi commented 3 years ago

@Amplifiyer I am facing this problem on both platforms (iOS and android) maybe not the same exception. I have also posted the logs in the question (Timed out waiting for subscription processor to start.). So after uninstalling the app from the device and re-running it, I navigate the user to the sign-in page. Once the user signs in, The below code is executed:-

  Future<void> signinUser() async {
    try {
      await InternetAddress.lookup(socketLookUpAddress);
      await Amplify.Auth.signIn(
        username: _username,
        password: _password,
      );
      await _initializeVariables(true);
    } on SocketException catch (_) {
      throw OfflineException();
    } on SessionUnavailableOfflineException catch (error) {
      throw error;
    } on AuthException catch (error) {
      throw error;
    } catch (error) {
      throw error;
    }
  }

_initializeVariables() initializes the variables:-

  Future<void> _initializeVariables([bool resetSessionValidity = false]) async {
    try {
      _authUser = await Amplify.Auth.getCurrentUser();
      _authSession = await Amplify.Auth.fetchAuthSession(
          options: CognitoSessionOptions(getAWSCredentials: true));
      userId = _authUser.userId;
      _userOffline = false;
      await _initializeUser();  <--- In this method I run datastore query
      await _initializeSharedPreferences(resetSessionValidity);
      notifyListeners();
    } on SessionExpiredException catch (error) {
      print(error.message);
      throw error;
    } on AuthException catch (error) {
      print('Error occured -> $error');
      throw error;
    } catch (error) {
      throw error;
    }

    notifyListeners();
  }

Datastore query

      final userResponse = await Amplify.DataStore.query(
        User.classType,
        where: User.ID.eq(_userId),
      );

It returns an empty list and so do all other APIs.

Amplifiyer commented 3 years ago

thanks @gauravdangi for more details. Your iOS logs will be helpful as well to see what's going on there. Also the code that you posted seems it's not using the hub's ready event. See this comment https://github.com/aws-amplify/amplify-flutter/issues/279#issuecomment-756409618 for more details on how to workaround the sync issue. Let us know if this workaround fixed any of your platforms.

gauravdangi commented 3 years ago

@Amplifiyer I have added Hub as a workaround when the user signs in or opens the app (auto-login). But I'm still facing this problem on android(working fine in iOS after setting up the Hub). Could you please confirm if I've implemented the hub the right way? It is not clear when we should listen to Hub. As given in the documentation, we should setup Hub Listener or Subscriber before calling Amplify.configure() to avoid missing some DataStore events. But while debugging I never saw this piece of code executing. But it gets executed if setup once the user logs in but it takes at least 20 to 40 seconds to get Datastore ready which is not a good user experience.

  Future<void> signinUser() async {
    try {
      await InternetAddress.lookup(socketLookUpAddress);
      await Amplify.Auth.signIn(
        username: _username,
        password: _password,
      );
      _syncDataStore();
      await _initializeVariables(true);
    } on SocketException catch (_) {
      throw OfflineException();
    } on SessionUnavailableOfflineException catch (error) {
      throw error;
    } on AuthException catch (error) {
      throw error;
    } catch (error) {
      throw error;
    }
  }

Listening to Datastore

  Future<void> _syncDataStore() async {
    try {
      await Amplify.DataStore.query(User.classType);
      _hubSubscription = Amplify.Hub.listen(
        [HubChannel.DataStore],
        (hubEvent) {
          if (hubEvent.eventName == 'ready') {
            _datastoreModelSynced = true;
            print(
                '-------------------------------DataStore is ready!---------------------------------');
            _initializeUser(true);
            _cancelListeningToHub();
          }
        },
      );
    } catch (error) {
      syncingToCloud = true;
      throw CustomCommonException(AppMessages.SyncError);
    }
  }

Android logs

E/amplify:aws-datastore(17394): Failure encountered while attempting to start API sync.
E/amplify:aws-datastore(17394): DataStoreException{message=Timed out waiting for subscription processor to start., cause=null, recoverySuggestion=Retry}
E/amplify:aws-datastore(17394):         at com.amplifyframework.datastore.syncengine.SubscriptionProcessor.startSubscriptions(SubscriptionProcessor.java:154)
E/amplify:aws-datastore(17394):         at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:309)
E/amplify:aws-datastore(17394):         at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$to6jsrpqq5TGk-8kfWyEK_7WfwQ.subscribe(Unknown Source:2)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletableSubscribeOn$SubscribeOnObserver.run(CompletableSubscribeOn.java:64)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
E/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
E/amplify:aws-datastore(17394):         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E/amplify:aws-datastore(17394):         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
E/amplify:aws-datastore(17394):         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
E/amplify:aws-datastore(17394):         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
E/amplify:aws-datastore(17394):         at java.lang.Thread.run(Thread.java:923)
W/amplify:aws-datastore(17394): API sync failed - transitioning to LOCAL_ONLY.
W/amplify:aws-datastore(17394): DataStoreException{message=Timed out waiting for subscription processor to start., cause=null, recoverySuggestion=Retry}
W/amplify:aws-datastore(17394):         at com.amplifyframework.datastore.syncengine.SubscriptionProcessor.startSubscriptions(SubscriptionProcessor.java:154)
W/amplify:aws-datastore(17394):         at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:309)
W/amplify:aws-datastore(17394):         at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$to6jsrpqq5TGk-8kfWyEK_7WfwQ.subscribe(Unknown Source:2)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.operators.completable.CompletableSubscribeOn$SubscribeOnObserver.run(CompletableSubscribeOn.java:64)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
W/amplify:aws-datastore(17394):         at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
W/amplify:aws-datastore(17394):         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W/amplify:aws-datastore(17394):         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
W/amplify:aws-datastore(17394):         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/amplify:aws-datastore(17394):         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/amplify:aws-datastore(17394):         at java.lang.Thread.run(Thread.java:923)
I/amplify:aws-datastore(17394): Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLY
I/amplify:aws-datastore(17394): Setting currentState to LOCAL_ONLY
I/amplify:aws-datastore(17394): Stopping subscription processor.
W/amplify:aws-api(17394): Thread interrupted waiting for connection acknowledgement
W/amplify:aws-api(17394): Thread interrupted waiting for connection acknowledgement
W/Choreographer(17394): Frame time is 0.087496 ms in the future!  Check that graphics HAL is generating vsync timestamps using the correct timebase.
I/amplify:aws-api(17394): No more active subscriptions. Closing web socket.
I/amplify:aws-datastore(17394): Stopped subscription processor.
D/AutoSessionTracker(17394): Activity paused: MainActivity
D/AutoSessionTracker(17394): Activity stopped: MainActivity
D/AutoSessionTracker(17394): Activity state saved: MainActivity
Application finished.

gauravdangi commented 3 years ago

@Amplifiyer @haverchuck Could you please tell me if there is a proper documentation of how we can set up a Hub listener as I'm not able to make it work for me. I'm in desperate need of help.

Amplifiyer commented 3 years ago

@gauravdangi, the problem doesn't seem to be the Hub listener (though the correct way would be to setup before calling configure()), but the Timeout exception that you are seeing in Android. What is the total number of records in your DDB table that DataStore is syncing up?

gauravdangi commented 3 years ago

@Amplifiyer There are a total of 5 tables each consisting of approx 20-30 records.

borainceler commented 3 years ago

I guess no one is using amplify for flutter, I am trying for a week now and I came across 50 different problems, I guess amplify is not really tested.

Anyway, I have the exact same problem, new project, new setup, and it throws the same error when trying to sync with dynamoDB but LOCAL_ONLY work.

might this sync issue be related to some issue with the Orchestrator Lambda? maybe the role assignment is not correct when done by the amplify built-in wizard output.

I will try to troubleshoot the lambda a bit, if not, I am out of amplify for another two until it matures to be usable. (I have other problems also but tried keep going)

Ashish-Nanda commented 3 years ago

@borainceler I'm sorry to hear that, this is not the experience we aim for when using Amplify.

Could you please share your schema and more details about your project setup for us to be able to better understand the possible root cause and reproduce the issue?

Also, when you say the amplify built-in wizard is that referring to the Admin UI?

Additionally I just wanted to let you know we are currently updating our docs to include a much more in-depth guide to getting started with DataStore that covers syncing to the cloud in more detail both with the CLI and Admin UI. We aim to have this go live by end of the week.

ErnestDaDev commented 3 years ago

@Ashish-Nanda I am having same issue, been three days debugging and trying out other ways to fix this. Everything works fine, schema has uploaded to dynamo, local storage works, datastore and API is enabled and pushing works with no errors. But once you add the api to sync online. it just says

E/amplify:aws-datastore( 7366): Failure encountered while attempting to start API sync. E/amplify:aws-datastore( 7366): DataStoreException{message=Timed out waiting for subscription processor to start., cause=null, recoverySuggestion=Retry} E/amplify:aws-datastore( 7366): at com.amplifyframework.datastore.syncengine.SubscriptionProcessor.startSubscriptions(SubscriptionProcessor.java:156) E/amplify:aws-datastore( 7366): at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:309) E/amplify:aws-datastore( 7366): at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$PVk58tU0K8ndPJYnH_tRmf4RGwE.subscribe(Unknown Source:2) E/amplify:aws-datastore( 7366): at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40) E/amplify:aws-datastore( 7366): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)

And once you try saving after failed initial setup, it still fails to sync data online.

I wanted to recommended AWS Amplify to the dev team but seems plugin is still full of bugs.

abelsantiagocortes commented 3 years ago

Same error tried everything and followed every tutorial. AWS datastore does not seem to be syncing with the API.

Ashish-Nanda commented 3 years ago

@BesavedTech @abelsantiagocortes Are you seeing this issue on both iOS and Android? Could you share your schema and if you are using the @auth directive?

Also do take a look at the suggestions in this comment to see if it helps resolve the issue https://github.com/aws-amplify/amplify-flutter/issues/279#issuecomment-756409618

If not we'll try to reproduce the behavior based on the answers to the above questions so that we can determine the root cause.

ErnestDaDev commented 3 years ago

@Ashish-Nanda I saw this on Android but I have gotten to sync to work now. I started a new flutter project and created a new app on the Admin UI, added the packages to pub spec, then used the url given from Admin UI to pull the initial setup. then i added amplify API with default settings and default schema then pushed to cloud.

Then added this on the main.dart

` bool amplifyConfigured = false;

Future _configureAmplify() async { AmplifyAuthCognito authPlugin = AmplifyAuthCognito(); AmplifyAnalyticsPinpoint analyticsPlugin = AmplifyAnalyticsPinpoint(); AmplifyStorageS3 storagePlugin = AmplifyStorageS3(); AmplifyDataStore datastorePlugin = AmplifyDataStore(modelProvider: ModelProvider.instance); AmplifyAPI datastoreAPI = AmplifyAPI();

try {
  await Amplify.addPlugins([
    authPlugin,
    storagePlugin,
    datastorePlugin,
    datastoreAPI,
    analyticsPlugin,
  ]);

  await Amplify.configure(amplifyconfig);
  setState(() {
    print("iscon");
    amplifyConfigured = true;
  });
  if (amplifyConfigured) {
    Amplify.Hub.listen([HubChannel.Auth], (hubEvent) {
      print("hubevent is ${hubEvent.eventName}");
      switch (hubEvent.eventName) {
        case "SIGNED_IN":
          {
            print("USER IS SIGNED IN");
          }
          break;
        case "SIGNED_OUT":
          {
            print("USER IS SIGNED OUT");
          }
          break;
        case "SESSION_EXPIRED":
          {
            print("USER IS SIGNED IN SESSION_EXPIRED");
          }
          break;
      }
    });

    Amplify.Hub.listen([HubChannel.DataStore], (hubEvent) {
      print("hubevent2 is ${hubEvent.eventName}");
    });
  }
} catch (e) {
  print("conf error is $e");
}

} `

Then it worked. I am still testing out the schema to see if my schema was an issue in terms of the structure or nested models. However I did realize that after creating schema and pushing to cloud. You must AMPLIFY PULL before you run your app again.

And now everything syncs to cloud awesomely.

peterBrxwn commented 3 years ago

I had this same timeout problem and reducing my data models to 4 (any value less than 5 should work) solved it. I think it has to do with GraphQL subscription default timeout period which is 10 seconds when there are less than 6 models.

see this https://github.com/aws-amplify/amplify-flutter/issues/615#issuecomment-860139959

HuiSF commented 3 years ago

Hi all, with the latest release of amplify-flutter (0.2.1) model subscription won't timeout on 3G and slower network. But please note that the ultimate result depends on device network condition. I would recommend to have a layer of handling on slow network to ensure better experience for your App users.

I will close this thread now, please feel free to follow up if you encounter any issues.

b-cancel commented 3 years ago

This is a bug report I just filed that includes a couple of bugs I found and the workarounds to each after reading your bug report I think it could help others that might land on this page https://github.com/aws-amplify/amplify-flutter/issues/822

ghost commented 2 years ago

DataStoreException(message: Failed to stop DataStore., recoverySuggestion: Retry., underlyingException: DataStoreException{message=Timed out acquiring orchestrator lock., cause=null, recoverySuggestion=Retry your request.})

aws-amplify / amplify-flutter

Retrieve data from Amplify Cloud back to DataStore #467