search

aws-amplify / amplify-flutter

A declarative library with an easy-to-use interface for building Flutter applications on AWS.
https://docs.amplify.aws
Apache License 2.0
1.31k stars 243 forks source link

DataStore sync issue from offline saved models (custom type fields) #2710

Closed 999Rub closed 1 year ago

999Rub commented 1 year ago

Description

This bug report is really close to https://github.com/aws-amplify/amplify-android/issues/1199 from the amplify-android.

I've got a GraphQL model (wich is provided in schema section) that I can save offline or online. When I use the app online, absolutely everything works fine.

When I launch the app online, then turn off the internet connectivity, create and save a new model, it is locally saved properly. Then I end up the program (still offline), turn back the internet and run the program again (online). The sync is processing, I can retrieve the data from my cloud, but the model that was saved in offline mode is not synced with the cloud.

It actually gives me a mix of DataStore and Amplify exception :

image

Regarding the Android issue mentioned above, I think my issue is really close to the same fix but I can't how to overpass it.

The thing that is really curious is, when I run the program, then turn off internet, create and save a model, and turn on internet (without closing the program this time), the model is correctly synced with my cloud.

I use the Auto Conflict resolver.

My Flutter doctor is fine.

If you need more details tell my and I will give you as much as I can.

Categories

Steps to Reproduce

  1. Use a template wich implement Datastore and AppSync with GraphQL API
  2. Create a model that has custom type fields (my custom type has only int and string fields)
  3. Run the program online
  4. Turn off internet
  5. Create and save an object with the model you previously created
  6. Close the program (offline)
  7. Turn the internet back
  8. Run the program
  9. Here it is

Screenshots

No response

Platforms

Flutter Version

3.7.3

Amplify Flutter Version

10.7.3

Deployment Method

Amplify CLI

Schema

# Surgery model is the one where the exception comes from (every custom fields are mentioned in the exception, I've tried to delete them one by one and this is not one field especially).

type Surgery @model @auth(rules: [{allow: owner}]) {
  id: ID!
  localAnesthesia: [LocalAnesthesia]!
  procedure: String
  physicalExamination: PhysicalExamination!
  checkList: CheckList!
  charts: Chart!
}

# The Chart model is an exemple of my custom type field (they all use basics types like int or string)

type Chart {
  timer: Int!
  lowChart: String!
  highChart: String!
}
HuiSF commented 1 year ago

Hi @999Rub did the console log the stack trace when this error happens? Could you share more logs? I'd like to identity where the exception originated in the code and further to know where to look.

999Rub commented 1 year ago

Here's all I've got:

W/amplify:aws-datastore(17941): DataStoreException{message=Error encountered while creating model schema, cause=DataStoreException{message=Failed to get fields for model., cause=AmplifyException{message=An invalid CustomType field was provided. charts must be an instance of SerializedCustomType or a List of instances of SerializedCustomType, cause=null, recoverySuggestion=Check if this model schema is a correct representation of the fields in the provided Object}, recoverySuggestion=Validate your model file.}, recoverySuggestion=See attached exception for more details}
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.appsync.AppSyncClient.create(AppSyncClient.java:147)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.syncengine.MutationProcessor.lambda$create$13$com-amplifyframework-datastore-syncengine-MutationProcessor(MutationProcessor.java:291)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.syncengine.MutationProcessor$$ExternalSyntheticLambda3.publish(Unknown Source:4)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.syncengine.MutationProcessor.lambda$publishWithStrategy$17(MutationProcessor.java:322)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.syncengine.MutationProcessor$$ExternalSyntheticLambda13.subscribe(Unknown Source:4)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.operators.single.SingleCreate.subscribeActual(SingleCreate.java:40)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap.subscribeActual(SingleFlatMap.java:37)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.operators.single.SingleDelayWithObservable$OtherSubscriber.onComplete(SingleDelayWithObservable.java:87)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.operators.single.SingleDelayWithObservable$OtherSubscriber.onNext(SingleDelayWithObservable.java:68)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.operators.observable.ObservableTimer$TimerObserver.run(ObservableTimer.java:67)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:41)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:28)
W/amplify:aws-datastore(17941):     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W/amplify:aws-datastore(17941):     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
W/amplify:aws-datastore(17941):     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/amplify:aws-datastore(17941):     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/amplify:aws-datastore(17941):     at java.lang.Thread.run(Thread.java:923)
W/amplify:aws-datastore(17941): Caused by: DataStoreException{message=Failed to get fields for model., cause=AmplifyException{message=An invalid CustomType field was provided. charts must be an instance of SerializedCustomType or a List of instances of SerializedCustomType, cause=null, recoverySuggestion=Check if this model schema is a correct representation of the fields in the provided Object}, recoverySuggestion=Validate your model file.}
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.buildCreationRequest(AppSyncRequestFactory.java:222)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.appsync.AppSyncClient.create(AppSyncClient.java:144)
W/amplify:aws-datastore(17941):     ... 18 more
W/amplify:aws-datastore(17941): Caused by: AmplifyException{message=An invalid CustomType field was provided. charts must be an instance of SerializedCustomType or a List of instances of SerializedCustomType, cause=null, recoverySuggestion=Check if this model schema is a correct representation of the fields in the provided Object}
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.extractCustomTypeFieldValue(AppSyncRequestFactory.java:547)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.extractFieldValue(AppSyncRequestFactory.java:500)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.extractFieldLevelData(AppSyncRequestFactory.java:427)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.getMapOfFieldNameAndValues(AppSyncRequestFactory.java:392)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.appsync.AppSyncRequestFactory.buildCreationRequest(AppSyncRequestFactory.java:219)
W/amplify:aws-datastore(17941):     ... 19 more

W/amplify:aws-datastore(17941): Error ended observation of mutation outbox:
W/amplify:aws-datastore(17941): DataStoreException{message=Failed to process java.lang.RuntimeException: DataStoreException{message=Error encountered while creating model schema, cause=DataStoreException{message=Failed to get fields for model., cause=AmplifyException{message=An invalid CustomType field was provided. charts must be an instance of SerializedCustomType or a List of instances of SerializedCustomType, cause=null, recoverySuggestion=Check if this model schema is a correct representation of the fields in the provided Object}, recoverySuggestion=Validate your model file.}, recoverySuggestion=See attached exception for more details}, cause=null, recoverySuggestion=Check your internet connection.}
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.syncengine.MutationProcessor.drainMutationOutbox(MutationProcessor.java:122)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.syncengine.MutationProcessor.lambda$startDrainingMutationOutbox$1$com-amplifyframework-datastore-syncengine-MutationProcessor(MutationProcessor.java:103)
W/amplify:aws-datastore(17941):     at com.amplifyframework.datastore.syncengine.MutationProcessor$$ExternalSyntheticLambda17.apply(Unknown Source:4)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.operators.observable.ObservableFlatMapCompletableCompletable$FlatMapCompletableMainObserver.onNext(ObservableFlatMapCompletableCompletable.java:97)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.drainNormal(ObservableObserveOn.java:201)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.operators.observable.ObservableObserveOn$ObserveOnObserver.run(ObservableObserveOn.java:255)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
W/amplify:aws-datastore(17941):     at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
W/amplify:aws-datastore(17941):     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W/amplify:aws-datastore(17941):     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
W/amplify:aws-datastore(17941):     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/amplify:aws-datastore(17941):     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/amplify:aws-datastore(17941):     at java.lang.Thread.run(Thread.java:923)

As I can see in the AmplifyDatastore.db that my model is correct (and in fact it is correct because its working with full online).

999Rub commented 1 year ago

The problem seems comes from the serialization of custom type in this Model, but I've tried to delete the 'charts' field as it's mentioned in the exception and the error still occurs on every custom types of this Model.

The curious thing is, when I try to create the same issue with another Model (with the exact custom types too), this is working without any exception.

When one of its model is saved offline and try to be synced at the restart of the program online and failed, every next future instances of this model will fail to sync (even if they're saved online this time).

I can't get why it is working on a model and not on the other one, while they have both custom types with basics field types (string, int, double, lists..).

I do an update of my sync expressions at runtime (sync expressions doesn't have any effect regarding the behavior when I comment it or not). However, when I do not use Amplify.DataStore.stop() and then Amplify.DataStore.start(), I don't receive any HubEvent if I previously saved a model offline (even those working), which I listen to for knowing when DataStore sent me ready and then switch from loading view to home page view.

When I only retrieve my data from the cloud (without offline saved before), and comment both Amplify.DataStore.stop() and Amplify.DataStore.start(), I've got the correct hubEvent in my listening method and everything works fine.

Maybe should I do not use Custom Types but general Models instead and just workaround with relation between each models ?

HuiSF commented 1 year ago

Thanks @999Rub the exception seems happening when clearing pending mutation box when your device is back online and deserializing stored data from the local DB (PendingMutationBox table rather than the actual Surgery table) I need to look into how the custom type data gets serialized and store in the PendingMutationBox table.

HuiSF commented 1 year ago

Quick follow up @999Rub which amplify-flutter version you are testing with?

999Rub commented 1 year ago

@HuiSF The amplify-flutter version is mentioned above but I use the 10.7.3 version. We are not in the same time zone but if you need more details I'm available as much as I can. If you can, give me a set of details you'll probably need when you can answer me and I'll give a set of details back (If you need too), it will avoid waiting for each other to wake up ^^. Will try to take a look about https://github.com/aws-amplify/amplify-flutter/issues/2710#issuecomment-1451139960 but I don't know whats happening under the hood in the PendingMutationBox so unfortunately I won't be the more efficient here...

The most mysterious thing to me (I don't know if you know why?), but why this issue occurs only on this model and not on the other ones (wich have custom types too) ?

HuiSF commented 1 year ago

Hi @999Rub sorry for the confusion, I believe 10.7.3 is the Amplify CLI version? Which you got from amplify --version? You should be able to find the amplify-flutter packages version in your pubspec.yaml.

999Rub commented 1 year ago

@HuiSF You right this is the Amplify CLI version, sorry for the misunderstanding, my actual version is amplify_flutter: ^0.6.8

999Rub commented 1 year ago

Hi @HuiSF , is there anything new from your side ?

HuiSF commented 1 year ago

Hi @999Rub sorry for the delay, I will dig into this issue today.

999Rub commented 1 year ago

Hi @999Rub sorry for the delay, I will dig into this issue today.

Don't worry, I have at least 2 or 3 weeks left to solve this issue. If there is no solution I will swap my custom types for classical models (I hope I don't have to do it). Just keep me up if you find something.

kalismeras61 commented 1 year ago

@HuiSF i have also same issue, i am using latest one

amplify_auth_cognito: 1.0.0-next.6 amplify_api: 1.0.0-next.6 amplify_flutter: ^1.0.0-next.6 amplify_core: 1.0.0-next.6 amplify_storage_s3: 1.0.0-next.6 amplify_datastore: ^1.0.0-next.6

vgribok commented 1 year ago

Yep, same here with the 1.0.0-next.6. It's an Android only problem. For me it happens when the phone is online. save() succeeds but up-link sync fails.

999Rub commented 1 year ago

Hi @HuiSF , I saw your PR and how the issue's fix evolve last month. What are the news/state about it ? Have you an estimated date of release for the fix ? This only for me, to know if I need to plan some changes in my app to avoid this problem before it goes for bêta or if a fix will be available at time.

Thanks for you work and time.

HuiSF commented 1 year ago

Hi @999Rub we are working with amplify-android maintainers to test and merge the PRs, will update the progress, thank you for your patience.

999Rub commented 1 year ago

Hi @HuiSF , I've seen that the open issue has been merged into main branch, thanks for you work and Amplify team's work. I think we can close this issue now.

Do you have any idea about when the fix will be available on the pub.dev ? Or maybe I don't have to wait that's available on pub.dev and simply pull the fix ?

Regards.

HuiSF commented 1 year ago

Hi @999Rub thanks for watching the progress! We are currently waiting for amplify-android releases, then we can integrate the fix into amplify-flutter. Amplify Android release should happen this week. We'll follow up ASAP once they release the new versions.

999Rub commented 1 year ago

Hi @HuiSF , I've seen a new update on pub dev for the v0 of amplify_flutter wich include your fix for this issue (from https://github.com/aws-amplify/amplify-flutter/releases/tag/v0.6.14 ), thanks for the work. Does the fix is also available on the latest v1 version (1.2.1) ? Or I need to wait until the v0 maintenance is done (19th July) to pull the fix ?

Regards.

HuiSF commented 1 year ago

Thank you for the feedback @999Rub ! And yes the fix is available in the latest v1 release v1.2.1.

Closing this issue 👍