Closed smvicky closed 2 years ago
smvicky
commented
2 years ago Hi @HuiSF This issue is only for Android devices, Datastore sync works properly on the iOS devices. So, this could be the issue with the plugin, please check the logs I have provided. I'm expecting quick resolution on this issue as we are stuck to rollout Android due to this issue.
haverchuck
commented
2 years ago @smvicky Do you know how many records from your DB should be syncing on this initial sync? Do you know how much time elapses before you get the exception? It looks like there is a connection timing out.
djorgji
commented
2 years ago @haverchuck The underlying dynamo DB has a lot of records (millions), however the specific model that is throwing the error has a sync expression that should cause it to never receive any records to sync, as there is no way in datastore to make it selectively syncing some tables, or how to prioritize the sync. The workaround we came up with looks like this:
DataStoreSyncExpression(
DeviceTelemetry.classType,
() =>
DeviceTelemetry.FOO.eq(null) &
DeviceTelemetry.FOO.ne(null),
),The above DataStoreSyncExpression works around the feature limitation and works perfectly in iOS but throws the exception above in Android.
We utilize this data separately through direct AppSync calls (after filtering based on FK it has to other data, ex: by deviceId) whenever we need it as it is only required when the user wants to perform specific analytics.
djorgji
commented
2 years ago Hi all, this is currently a blocker for our android release. Any updates?
P.S. @smvicky and I work on the same team facing this issue.
HuiSF
commented
2 years ago Hi @djorgji sorry for the delayed follow up.
From the exception logs posted by @smvicky that looks like the GraphQL sync query for model DeviceTelemetry timed out. I may need more info to help triaging this issue as I couldn't reproduce it by using similar sync expression.
DeviceTelemetry?<path-to-your-flutter-project>/android)You can view the details of the underlying HTTP requests. e.g.
djorgji
commented
2 years ago @HuiSF I am going to let @smvicky do the profiling you asked. However after some further testing it looks like it is a combination of conditions.
The exception is thrown when a:
Our desire is to not sync any records on these tables. Changing lt to gt bellow (or removing the and completely) "fixes" the exception, but it is not what we want as we experience unacceptable performance on tables we want to sync. While also syncing data that we will never use, bloating our mobile device data costs and storage footprint.
final date = DateFormat('yyyy-MM-dd Hms').format(DateTime(2021));
DeviceTelemetry.classType,
() => DeviceTelemetry.ACCOUNTID
.eq(SyncParams.accountID) // tenant discriminator column
.and(DeviceTelemetry.EVENTDATE.lt(date)) // no records older than Jan 2022Data model:
type DeviceTelemetry @model @auth(rules: [{allow: custom}]) {
id: ID!
temperature: Measurement
humidity: Measurement
barometricPressure: Measurement
deviceTelemetryDeviceId: ID @index(name: "telemetryByDate", queryField: "telemetryByDate", sortKeyFields: ["eventDate"])
device: Device @hasOne (fields: ["deviceTelemetryDeviceId"])
eventDate: AWSDateTime
accountID: String
}
type Measurement {
label: String!
value: Float!
uom: String!
}Providing a way to disable sync on some tables without the sync expression workaround would work great to solve our issue as well. Ideally also being able to tell datastore the sync priority of tables would also be nice to allow developer to optimize to their specific use cases.
Again this is currently only an issue on Android, iOS seems to be okay with this workaround when selectively syncing.
HuiSF
commented
2 years ago Hi @djorgji thanks for the follow up and details of debugging.
It sounds like your syncExpression created a sync query that takes long time for DynamoDB to scan through records, which may cause timeout exception.
Please take a look at my comment in an issue asking about a similar use case.
In addition to the linked comments, if you want to use syncExpression on fields for this kind of use case, you would need to set up SDI on this fields. Please refer to this document.
djorgji
commented
2 years ago @HuiSF I tried what you said in your other comment in a quick test and I got this exception:
DataStoreSyncExpression(DeviceTelemetry.classType, () => DeviceTelemetry.ID.eq(null))
---
W/amplify:aws-datastore(13336): Caused by: io.reactivex.rxjava3.exceptions.CompositeException$ExceptionOverview: Multiple exceptions (2)
W/amplify:aws-datastore(13336): |-- com.amplifyframework.datastore.DataStoreException: Failure performing sync query to AppSync: [GraphQLResponse.Error{message='The variables input contains a field name 'deviceTelemetry.id' that is not defined for input object type 'ModelDeviceTelemetryFilterInput' ', locations='null', path='null', extensions='null'}]
W/amplify:aws-datastore(13336): at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$sync$0(AppSyncClient.java:113)
W/amplify:aws-datastore(13336): |-- java.lang.RuntimeException: Methods marked with @UiThread must be executed on the main thread. Current thread: RxCachedThreadScheduler-35
W/amplify:aws-datastore(13336): at io.flutter.embedding.engine.FlutterJNI.ensureRunningOnMainThread(FlutterJNI.java:1405)
I/amplify:aws-datastore(13336): Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLYThe model does have id: ID! in it see above.
I will try the SDI approach later as a workaround,
However this has to be a temporary workload as in our use case the tenancy is part of the sync expression, sending data to client that should not have the data is not secure even ignoring the extra data sent to the device.
HuiSF
commented
2 years ago Thanks for the follow up @djorgji
The variables input contains a field name 'deviceTelemetry.id'This deviceTelemetry.id looks wrong to me, it should just be id.
Can you paste the generate model file of DeviceTelemetry?
Also this file schema.graphql in <your-flutter-project>/amplify/backend/<your-api-name>/build/schema.graphql.
djorgji
commented
2 years ago @HuiSF Thanks for the reply, I changed the file manually to id for some reason the model generator had put the wrong thing there.
static final QueryField ID = QueryField(fieldName: "deviceTelemetry.id");
//changed manually to
static final QueryField ID = QueryField(fieldName: "id");
//regenerating the model goes back to
static final QueryField ID = QueryField(fieldName: "deviceTelemetry.id");However after the manual change I am still getting:
W/amplify:aws-datastore(20165): DataStoreException{message=Initial sync during DataStore initialization failed., cause=io.reactivex.rxjava3.exceptions.CompositeException: 2 exceptions occurred. , recoverySuggestion=There is a possibility that there is a bug if this error persists. Please take a look at
W/amplify:aws-datastore(20165): https://github.com/aws-amplify/amplify-android/issues to see if there are any existing issues that
W/amplify:aws-datastore(20165): match your scenario, and file an issue with the details of the bug if there isn't.}
W/amplify:aws-datastore(20165): at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:311)
W/amplify:aws-datastore(20165): at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$PVk58tU0K8ndPJYnH_tRmf4RGwE.subscribe(Unknown Source:2)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.completable.CompletableSubscribeOn$SubscribeOnObserver.run(CompletableSubscribeOn.java:64)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
W/amplify:aws-datastore(20165): at java.util.concurrent.FutureTask.run(FutureTask.java:266)
W/amplify:aws-datastore(20165): at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
W/amplify:aws-datastore(20165): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/amplify:aws-datastore(20165): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/amplify:aws-datastore(20165): at java.lang.Thread.run(Thread.java:923)
W/amplify:aws-datastore(20165): Caused by: io.reactivex.rxjava3.exceptions.CompositeException: 2 exceptions occurred.
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek$CompletableObserverImplementation.onError(CompletablePeek.java:92)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapCompletable$FlatMapCompletableObserver.onError(SingleFlatMapCompletable.java:97)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:117)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.util.AtomicThrowable.tryTerminateConsumer(AtomicThrowable.java:156)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapCompletable$ConcatMapCompletableObserver.onError(FlowableConcatMapCompletable.java:129)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.util.AtomicThrowable.tryTerminateConsumer(AtomicThrowable.java:94)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.util.HalfSerializer.onError(HalfSerializer.java:67)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapImmediate.onError(FlowableConcatMap.java:199)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$ObserveOnSubscriber.runBackfused(FlowableObserveOn.java:449)
W/amplify:aws-datastore(20165): at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$BaseObserveOnSubscriber.run(FlowableObserveOn.java:172)
W/amplify:aws-datastore(20165): ... 7 more
E/amplify:aws-datastore(20165): |-- java.lang.RuntimeException: Methods marked with @UiThread must be executed on the main thread. Current thread: RxCachedThreadScheduler-8
W/amplify:aws-datastore(20165): Caused by: io.reactivex.rxjava3.exceptions.CompositeException$ExceptionOverview: Multiple exceptions (2)
W/amplify:aws-datastore(20165): |-- com.amplifyframework.datastore.DataStoreException: Failure performing sync query to AppSync.
W/amplify:aws-datastore(20165): at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$sync$1(AppSyncClient.java:121)
W/amplify:aws-datastore(20165): |-- com.amplifyframework.api.ApiException: OkHttp client request failed.
W/amplify:aws-datastore(20165): at com.amplifyframework.api.aws.AppSyncGraphQLOperation$OkHttpCallback.onFailure(AppSyncGraphQLOperation.java:161)
W/amplify:aws-datastore(20165): |-- java.net.SocketTimeoutException: timeout
W/amplify:aws-datastore(20165): at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:677)
W/amplify:aws-datastore(20165): |-- java.lang.RuntimeException: Methods marked with @UiThread must be executed on the main thread. Current thread: RxCachedThreadScheduler-8
W/amplify:aws-datastore(20165): at io.flutter.embedding.engine.FlutterJNI.ensureRunningOnMainThread(FlutterJNI.java:1405)
I/amplify:aws-datastore(20165): Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLYI can repro this exactly same error in Android. But the same sync expression works correctly in iOS, so it looks like a bug to me. But this is not directly related to the original issue described in this thread.
To summarize the issue:
Next steps:
Hi @djorgji to clarify your case
Our desire is to not sync any records on these tables.
Take DeviceTelemetry model, do you need to use DataStore APIs to manipulate data of this model? Including
If you are not expecting any of DataStore specific feature to use with this model, you can simple remove the reference of this model from ModelProvider.dart. And when you need to create a new record of this model, you can use API plugin to make a GraphQL request using GraphQL helper with the generated model.
djorgji
commented
2 years ago
- The original issue descried in this thread is mostly like caused by slow sync query operation
- To achieve what you desired, there is a workaround may work, but this workaround doesn't work in Android due a bug
Correct there are a couple of tables that hold a lot of data, that we want to stop from sync through the data store but we want to still have access through the GraphQL API.
Take
DeviceTelemetrymodel, do you need to use DataStore APIs to manipulate data of this model? Including
- offline data persistence
- sync data to DynamoDB
If you are not expecting any of DataStore specific feature to use with this model, you can simple remove the reference of this model from
ModelProvider.dart. And when you need to create a new record of this model, you can use API plugin to make a GraphQL request using GraphQL helper with the generated model.
How would that work? To initialize the API I need to pass in the ModelProvider Amplify.addPlugin(AmplifyAPI(modelProvider: ModelProvider.instance));. If I remove the DeviceTelemetry from the ModelProvider.dart file I wont be able to to use the API to query it?
djorgji
commented
2 years ago @HuiSF Hi, so I did some more testing in iOS for the different sync expressions, there is still a timeout in iOS (including when using DataStoreSyncExpression(DeviceTelemetry.classType, () => DeviceTelemetry.ID.eq(null))). However iOS continues retrying without transitioning the to LOCAL_ONLY, unlike Android that doesI/amplify:aws-datastore(20359): Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLY right away when the timeout occurs. Is there away to get the two to be more consistent?
flutter: DATASTORE_HUB_EVENT - modelSynced
[InitialSyncOperation] Beginning sync for DeviceTelemetry
DataStoreError: An error occurred syncing DeviceTelemetry
Caused by:
DataStoreError: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Execution timed out.", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncDeviceTelemetries")]), extensions: Optional(["errorInfo": Amplify.JSONValue.null, "errorType": Amplify.JSONValue.string("ExecutionTimeout"), "data": Amplify.JSONValue.null]))]
Recovery suggestion: The list of `GraphQLError` contains service-specific messages
Caused by:
GraphQLResponseError<PaginatedList<AnyModel>>: GraphQL service returned a successful response containing errors: [Amplify.GraphQLError(message: "Execution timed out.", locations: Optional([Amplify.GraphQLError.Location(line: 2, column: 3)]), path: Optional([Amplify.JSONValue.string("syncDeviceTelemetries")]), extensions: Optional(["errorInfo": Amplify.JSONValue.null, "errorType": Amplify.JSONValue.string("ExecutionTimeout"), "data": Amplify.JSONValue.null]))]
Recovery suggestion: The list of `GraphQLError` contains service-specific messages
[IncomingAsyncSubscriptionEventToAnyModelMapper] Received completion: finished
flutter: DATASTORE_HUB_EVENT - modelSyncedThe above timeout occurs event when using ID.eq(null) which I assume is indexed, and the most "efficient" way to run the query. I don't think utilizing this workaround will be sufficient. I can see others hitting this issue with scalability as their apps grow, or as data accumulates. The Datastore API needs to support a way to not sync some tables.
HuiSF
commented
2 years ago Hi @djorgji sorry I missed tracking of your comments somehow.
Correct there are a couple of tables that hold a lot of data
How would that work? To initialize the API I need to pass in the ModelProvider
This sounds critical to your use case, I think we should remove the model from DataStore syncing process completely. I understand you have to also use ModelProvider with API plugin, here's a workaround:
ModelProvider, rename it to ModelProviderAPI, for example to use with the API pluginModelProvider remove the references of model that you want to prevent syncing within DataStoreModelProviderdjorgji
commented
2 years ago Hi @HuiSF sorry for the radio silence for this long, we have removed our dependency to Datastore as the possible configurations to the datastore did not fit our IoT use case that generates a lot of data and update events all the time. We are just utilizing the API for the time being and working towards rolling out own solution utilizing Hive.
However we have hit a couple of snugs utilizing the Graphql API Helper:
1) It looks like it does not properly put the version in delete mutations as we are hitting this:
{
"data": {
"deleteNotifications": null
},
"errors": [
{
"path": [
"deleteNotifications"
],
"data": {
"_deleted": null,
"_version": 2,
"id": "23d847d4-eb27-43ec-90c1-9be6cc085122"
},
"errorType": "ConflictUnhandled",
"errorInfo": null,
"locations": [
{
"line": 4,
"column": 3,
"sourceName": null
}
],
"message": "Conflict resolver rejects mutation."
}
]
}2) The generated model code does not include _version. Without the _version parameter the update/delete mutations would not work.
3) Same things as above with the _deleted flag.
4) The generated model code also does not properly copy updatedAt, and createdAt, when using the copyWith methods causing issues when trying to use those fields after calling copyWith
The generation issues we have fixed by manually editing the code, but it would be nice not to have to worry about our changes every time we generate. The delete issue we are working on resolving by utilizing this, and writing out own graphql statements.
HuiSF
commented
2 years ago Hello @djorgji Thanks for following up.
we have removed our dependency to Datastore as the possible configurations to the datastore did not fit our IoT use case that generates a lot of data and update events all the time
Sorry to hear about this. I've heard AppSync subscription now supports runtime filter that functioning on the service side, hopefully this feature can come to DataStore soon, that may help with this type of use case.
However we have hit a couple of snugs utilizing the Graphql API Helper
_version, _deleted and _lastChangedAt are hidden mandatory fields added to GraphQL schema when conflict detection is enabled for the API category. These fields are not accessible in the codegen generated model. Hence, they are missing from the GraphQL documents generated using GraphQL helper.
Conflict detection is mandatory for DataStore function, if you are not using DataStore at all, you can consider to turn off conflict detection.
djorgji
commented
2 years ago Hi @HuiSF,
Even if we are not utilizing the Amplify Datastore directly we need to be able to utilize conflict resolution for own development when utilizing the API Helper. It is part of the API category that the Datastore utilizes, not part of the Datastore. As of right now the delete/update mutations through the API Helper library will not work at all since there is no _version.
I understand you are trying to help developers not to get confused, or to screw things up, but this is very limiting for developers that have use cases where the Datastore does not fit, or know that they are doing. Giving us some more freedom would allow us to optimize for our specific use cases, while utilizing amplify as a starter package that provides a lot of conveniences. I don't want to throw the baby out with the bathwater. It does not have to be part of the default behavior, I would be happy even if it was "hidden" behind some command flag, or schema config.
I think it is a requirement to be able to utilize what the API category provides fully (including conflict resolution) through just the GraphQL API (or the helper library) even if I am not utilizing the Datastore, ex: in Lambdas, or third party system integrations, or in UIs without caching.
Same thing for _deleted which is not something we can use in filters through the API, which forces us to get more data from the server than we want, and then filter them locally. I would like to keep the soft delete capabilities without having to reimplement them myself.
Finally, disabling the conflict strategy disables the Amplify Studio Content management capabilities that we currently utilize for our support staff.
HuiSF
commented
2 years ago Thanks for the follow up @djorgji I understood the pain points. Better support for this use case is under investigation now, and we have an issue for tracking the progress.
As our discussion now shifted from the very original issue of this thread, do you mind if I close this issue, and we will be updating the process in above linked issue.
djorgji
commented
2 years ago I would be ok with closing this if we add _deleted filter support to that other issue.
HuiSF
commented
2 years ago Thanks @djorgji I've created this issue (point 2 lists your use case) to track on this feature request. Please feel free to follow up.
danfreid
commented
1 year ago May be irrelevant for the original poster, but for the first use case, we deployed a pre-push hook script to explicitly set the "lastSync" to a large number and keep certain models from being processed for DataStore syncs. This allows us to use DataStore for the majority of the models but ignore high latency for those that we never want to get to the client (for us, AuditLog, SystemEvents, DailyActiveUser, etc.)
Also, there is one model that has millions of rows that we DO want some on the client, so we created our own multi-threaded hydration approach to get large data sets to the client (and then subscribe to an id-only projection for updates.) Like you, we want the benefits of DataStore without the limitations, but there is usually a way around the shortfalls.
Description
Getting following Exception on Amplify.Datastore.Start()
Categories
Steps to Reproduce
No response
Screenshots
No response
Platforms
Environment
Dependencies
Device
All Android devices
OS
All vesions
CLI Version
7.6.26
Additional Context
I/amplify:aws-datastore(28141): Successfully sync'd down model state from cloud. I/amplify:aws-datastore(28141): Successfully sync'd down model state from cloud. W/amplify:aws-datastore(28141): Initial cloud sync failed for DeviceTelemetry. W/amplify:aws-datastore(28141): DataStoreException{message=Failure performing sync query to AppSync., cause=ApiException{message=OkHttp client request failed., cause=java.net.SocketTimeoutException: timeout, recoverySuggestion=See attached exception for more details.}, recoverySuggestion=Sorry, we don't have a suggested fix for this error yet.} W/amplify:aws-datastore(28141): at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$sync$1(AppSyncClient.java:121) W/amplify:aws-datastore(28141): at com.amplifyframework.datastore.appsync.-$$Lambda$AppSyncClient$7jOsUMJbab5VI0cCjZ1sP3zQNIU.accept(Unknown Source:4) W/amplify:aws-datastore(28141): at com.amplifyframework.api.aws.AppSyncGraphQLOperation$OkHttpCallback.onFailure(AppSyncGraphQLOperation.java:161) W/amplify:aws-datastore(28141): at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:525) W/amplify:aws-datastore(28141): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) W/amplify:aws-datastore(28141): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) W/amplify:aws-datastore(28141): at java.lang.Thread.run(Thread.java:764) W/amplify:aws-datastore(28141): Caused by: ApiException{message=OkHttp client request failed., cause=java.net.SocketTimeoutException: timeout, recoverySuggestion=See attached exception for more details.} W/amplify:aws-datastore(28141): ... 5 more W/amplify:aws-datastore(28141): Caused by: java.net.SocketTimeoutException: timeout W/amplify:aws-datastore(28141): at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:677) W/amplify:aws-datastore(28141): at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.kt:686) W/amplify:aws-datastore(28141): at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:143) W/amplify:aws-datastore(28141): at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:96) W/amplify:aws-datastore(28141): at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106) W/amplify:aws-datastore(28141): at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79) W/amplify:aws-datastore(28141): at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) W/amplify:aws-datastore(28141): at com.amplifyframework.api.aws.UserAgentInterceptor.intercept(UserAgentInterceptor.java:56) W/amplify:aws-datastore(28141): at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) W/amplify:aws-datastore(28141): at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) W/amplify:aws-datastore(28141): at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) W/amplify:aws-datastore(28141): at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) W/amplify:aws-datastore(28141): at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) W/amplify:aws-datastore(28141): at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) W/amplify:aws-datastore(28141): at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) W/amplify:aws-datastore(28141): at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) W/amplify:aws-datastore(28141): at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) W/amplify:aws-datastore(28141): at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) W/amplify:aws-datastore(28141): at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) W/amplify:aws-datastore(28141): ... 3 more E/amplify:aws-datastore(28141): Failure encountered while attempting to start API sync. E/amplify:aws-datastore(28141): DataStoreException{message=Initial sync during DataStore initialization failed., cause=io.reactivex.rxjava3.exceptions.CompositeException: 2 exceptions occurred. , recoverySuggestion=There is a possibility that there is a bug if this error persists. Please take a look at E/amplify:aws-datastore(28141): https://github.com/aws-amplify/amplify-android/issues to see if there are any existing issues that E/amplify:aws-datastore(28141): match your scenario, and file an issue with the details of the bug if there isn't.} E/amplify:aws-datastore(28141): at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:327) E/amplify:aws-datastore(28141): at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$PVk58tU0K8ndPJYnH_tRmf4RGwE.subscribe(Unknown Source:2) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletableSubscribeOn$SubscribeOnObserver.run(CompletableSubscribeOn.java:64) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56) E/amplify:aws-datastore(28141): at java.util.concurrent.FutureTask.run(FutureTask.java:266) E/amplify:aws-datastore(28141): at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301) E/amplify:aws-datastore(28141): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) E/amplify:aws-datastore(28141): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) E/amplify:aws-datastore(28141): at java.lang.Thread.run(Thread.java:764) E/amplify:aws-datastore(28141): Caused by: io.reactivex.rxjava3.exceptions.CompositeException: 2 exceptions occurred. E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek$CompletableObserverImplementation.onError(CompletablePeek.java:92) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapCompletable$FlatMapCompletableObserver.onError(SingleFlatMapCompletable.java:97) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:117) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.util.AtomicThrowable.tryTerminateConsumer(AtomicThrowable.java:156) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapCompletable$ConcatMapCompletableObserver.onError(FlowableConcatMapCompletable.java:129) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.util.AtomicThrowable.tryTerminateConsumer(AtomicThrowable.java:94) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.util.HalfSerializer.onError(HalfSerializer.java:67) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapImmediate.onError(FlowableConcatMap.java:199) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$ObserveOnSubscriber.runBackfused(FlowableObserveOn.java:449) E/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$BaseObserveOnSubscriber.run(FlowableObserveOn.java:172) E/amplify:aws-datastore(28141): ... 7 more E/amplify:aws-datastore(28141): Caused by: io.reactivex.rxjava3.exceptions.CompositeException$ExceptionOverview: Multiple exceptions (2) E/amplify:aws-datastore(28141): |-- com.amplifyframework.datastore.DataStoreException: Failure performing sync query to AppSync. E/amplify:aws-datastore(28141): at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$sync$1(AppSyncClient.java:121) E/amplify:aws-datastore(28141): |-- com.amplifyframework.api.ApiException: OkHttp client request failed. E/amplify:aws-datastore(28141): at com.amplifyframework.api.aws.AppSyncGraphQLOperation$OkHttpCallback.onFailure(AppSyncGraphQLOperation.java:161) E/amplify:aws-datastore(28141): |-- java.net.SocketTimeoutException: timeout E/amplify:aws-datastore(28141): at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:677) E/amplify:aws-datastore(28141): |-- java.lang.RuntimeException: Methods marked with @UiThread must be executed on the main thread. Current thread: RxCachedThreadScheduler-4 E/amplify:aws-datastore(28141): at io.flutter.embedding.engine.FlutterJNI.ensureRunningOnMainThread(FlutterJNI.java:1405) W/amplify:aws-datastore(28141): API sync failed - transitioning to LOCAL_ONLY. W/amplify:aws-datastore(28141): DataStoreException{message=Initial sync during DataStore initialization failed., cause=io.reactivex.rxjava3.exceptions.CompositeException: 2 exceptions occurred. , recoverySuggestion=There is a possibility that there is a bug if this error persists. Please take a look at W/amplify:aws-datastore(28141): https://github.com/aws-amplify/amplify-android/issues to see if there are any existing issues that W/amplify:aws-datastore(28141): match your scenario, and file an issue with the details of the bug if there isn't.} W/amplify:aws-datastore(28141): at com.amplifyframework.datastore.syncengine.Orchestrator.lambda$startApiSync$3$Orchestrator(Orchestrator.java:327) W/amplify:aws-datastore(28141): at com.amplifyframework.datastore.syncengine.-$$Lambda$Orchestrator$PVk58tU0K8ndPJYnH_tRmf4RGwE.subscribe(Unknown Source:2) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:40) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletableSubscribeOn$SubscribeOnObserver.run(CompletableSubscribeOn.java:64) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56) W/amplify:aws-datastore(28141): at java.util.concurrent.FutureTask.run(FutureTask.java:266) W/amplify:aws-datastore(28141): at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301) W/amplify:aws-datastore(28141): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) W/amplify:aws-datastore(28141): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) W/amplify:aws-datastore(28141): at java.lang.Thread.run(Thread.java:764) W/amplify:aws-datastore(28141): Caused by: io.reactivex.rxjava3.exceptions.CompositeException: 2 exceptions occurred. W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek$CompletableObserverImplementation.onError(CompletablePeek.java:92) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapCompletable$FlatMapCompletableObserver.onError(SingleFlatMapCompletable.java:97) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:117) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.util.AtomicThrowable.tryTerminateConsumer(AtomicThrowable.java:156) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapCompletable$ConcatMapCompletableObserver.onError(FlowableConcatMapCompletable.java:129) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.util.AtomicThrowable.tryTerminateConsumer(AtomicThrowable.java:94) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.util.HalfSerializer.onError(HalfSerializer.java:67) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapImmediate.onError(FlowableConcatMap.java:199) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$ObserveOnSubscriber.runBackfused(FlowableObserveOn.java:449) W/amplify:aws-datastore(28141): at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$BaseObserveOnSubscriber.run(FlowableObserveOn.java:172) W/amplify:aws-datastore(28141): ... 7 more W/amplify:aws-datastore(28141): Caused by: io.reactivex.rxjava3.exceptions.CompositeException$ExceptionOverview: Multiple exceptions (2) W/amplify:aws-datastore(28141): |-- com.amplifyframework.datastore.DataStoreException: Failure performing sync query to AppSync. W/amplify:aws-datastore(28141): at com.amplifyframework.datastore.appsync.AppSyncClient.lambda$sync$1(AppSyncClient.java:121) W/amplify:aws-datastore(28141): |-- com.amplifyframework.api.ApiException: OkHttp client request failed. W/amplify:aws-datastore(28141): at com.amplifyframework.api.aws.AppSyncGraphQLOperation$OkHttpCallback.onFailure(AppSyncGraphQLOperation.java:161) W/amplify:aws-datastore(28141): |-- java.net.SocketTimeoutException: timeout W/amplify:aws-datastore(28141): at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:677) W/amplify:aws-datastore(28141): |-- java.lang.RuntimeException: Methods marked with @UiThread must be executed on the main thread. Current thread: RxCachedThreadScheduler-4 W/amplify:aws-datastore(28141): at io.flutter.embedding.engine.FlutterJNI.ensureRunningOnMainThread(FlutterJNI.java:1405) I/amplify:aws-datastore(28141): Orchestrator transitioning from SYNC_VIA_API to LOCAL_ONLY I/amplify:aws-datastore(28141): Setting currentState to LOCAL_ONLY I/amplify:aws-datastore(28141): Stopping subscription processor. I/amplify:aws-datastore(28141): Stopped subscription processor.