Data not showing even though data is in datastore

[amruth-movano]

Describe the bug

Datastore not fetching data even though it is in dynamodb

Steps To Reproduce

- 
Fetch the data
For few days it is showing data and for few it is not

I am having data at backend on each day from 5th to 11th May. But query not returning anything.

Please let us know if anything need to be changed at backend or front end or its from datastore issue

Expected behavior

It should show the data if it is present in dynamodb.

Amplify Framework Version

2.33.3

Amplify Categories

DataStore

Dependency manager

Swift PM

Swift version

5.9.2

CLI version

12.11.1

Xcode version

15.0

Relevant log output

select
  "root"."@@primaryKey" as "@@primaryKey", "root"."user_id" as "user_id", "root"."time_stamp" as "time_stamp",
  "root"."breath_rate" as "breath_rate", "root"."calories" as "calories", "root"."createdAt" as "createdAt",
  "root"."data_timezone" as "data_timezone", "root"."distance_covered" as "distance_covered", "root"."heart_rate" as "heart_rate",
  "root"."hrv" as "hrv", "root"."owner" as "owner", "root"."ring_serial_no" as "ring_serial_no",
  "root"."skin_temperature" as "skin_temperature", "root"."spo2" as "spo2", "root"."step_count" as "step_count",
  "root"."updatedAt" as "updatedAt"
from "TimedData" as "root"
where 1 = 1
  and (
    "root"."user_id" = '7***********************4'
    and "root"."time_stamp" between '2024-05-04T18:30:00.000Z' and '2024-05-11T18:29:59.000Z'
  )
order by "root"."time_stamp" asc
select
  "root"."@@primaryKey" as "@@primaryKey", "root"."user_id" as "user_id", "root"."time_stamp" as "time_stamp",
  "root"."breath_rate" as "breath_rate", "root"."calories" as "calories", "root"."createdAt" as "createdAt",
  "root"."data_timezone" as "data_timezone", "root"."distance_covered" as "distance_covered", "root"."heart_rate" as "heart_rate",
  "root"."hrv" as "hrv", "root"."owner" as "owner", "root"."ring_serial_no" as "ring_serial_no",
  "root"."skin_temperature" as "skin_temperature", "root"."spo2" as "spo2", "root"."step_count" as "step_count",
  "root"."updatedAt" as "updatedAt"
from "TimedData" as "root"
where 1 = 1
  and (
    "root"."user_id" = '7***************'
    and "root"."time_stamp" between '2024-05-04T18:30:00.000Z' and '2024-05-11T18:29:59.000Z'
  )
order by "root"."time_stamp" asc

Is this a regression?

Yes

Regression additional context

No response

Platforms

iOS

OS Version

iOS 17

Device

iPhone 15 Plus

Specific to simulators

No response

Additional context

No response

[thisisabhash]

Thank you for posting this.

• Can you share the verbose logs and code snippets in the ticket?
• Are you seeing errors while syncing data from AppSync?

[amruth-movano]

@thisisabhash Verbose logs -

select "root"."@@primaryKey" as "@@primaryKey", "root"."user_id" as "user_id", "root"."time_stamp" as "time_stamp", "root"."breath_rate" as "breath_rate", "root"."calories" as "calories", "root"."createdAt" as "createdAt", "root"."data_timezone" as "data_timezone", "root"."distance_covered" as "distance_covered", "root"."heart_rate" as "heart_rate", "root"."hrv" as "hrv", "root"."owner" as "owner", "root"."ring_serial_no" as "ring_serial_no", "root"."skin_temperature" as "skin_temperature", "root"."spo2" as "spo2", "root"."step_count" as "step_count", "root"."updatedAt" as "updatedAt" from "TimedData" as "root" where 1 = 1 and ( "root"."user_id" = '74' and "root"."time_stamp" between '2024-05-04T18:30:00.000Z' and '2024-05-11T18:29:59.000Z' ) order by "root"."time_stamp" asc select "root"."@@primaryKey" as "@@primaryKey", "root"."user_id" as "user_id", "root"."time_stamp" as "time_stamp", "root"."breath_rate" as "breath_rate", "root"."calories" as "calories", "root"."createdAt" as "createdAt", "root"."data_timezone" as "data_timezone", "root"."distance_covered" as "distance_covered", "root"."heart_rate" as "heart_rate", "root"."hrv" as "hrv", "root"."owner" as "owner", "root"."ring_serial_no" as "ring_serial_no", "root"."skin_temperature" as "skin_temperature", "root"."spo2" as "spo2", "root"."step_count" as "step_count", "root"."updatedAt" as "updatedAt" from "TimedData" as "root" where 1 = 1 and ( "root"."user_id" = '74' and "root"."time_stamp" between '2024-05-04T18:30:00.000Z' and '2024-05-11T18:29:59.000Z' ) order by "root"."time_stamp" asc

And not getting any errors also

[thisisabhash]

Can you enable verbose logs like this

Amplify.Logging.logLevel = .verbose

and then start your app and call relevant DataStore APIs? Datastore.start() / Datastore.query()? Does it return an empty list? or does it throw an error?

[amruth-movano]

@thisisabhash Yes, it is already enabled.

[amruth-movano]

@thisisabhash please see the whole verbose after doing Datastore.start() -

verbose.txt

[5d]

Hi @amruth-movano ,

Could you share how you are configuring the datastore plugin?

The Datastore.query performs local queries against a local SQL database, which contains only the data that has been synced. This process depends on how you have defined your sync predicate and the configuration options for your local datastore, such as the maximum number of synced records.

Since it's impractical to store all server-side data locally, you might need to use an API.query to fetch records from your remote AppSync server with predicate for old data, especially for those that exceed the maximum synced records stored locally for the initial full sync.

[amruth-movano]

Hi @5d Please see attached files. We are calling startAndPopulateDataStore from DataStoreManager class, whenever users logins or launch the app. Also maximum sync records limit is 15000. And for all data to fetch we are using datastore fetch query like -

public func fetchTimedData(from start: Date, to end: Date) async throws -> [DataStore.TimedData] { let userId = try await userId() let results = try await dataStore.query( TimedData.self, where: TimedData.keys.user_id.eq(userId).and(TimedData.keys.time_stamp.between( start: start.temporalDateTime, end: end.temporalDateTime )), sort: nil, paginate: nil ) return results.compactMap(TimedDataMapper.map) }

amplify.zip

[amruth-movano]

@thisisabhash @5d Also I want to know, for our users in India Timezone, data is populating very fast but in US timezone data population is taking so much time.

[5d]

Hi @amruth-movano ,

What is the typical or average volume/amount of data that is stored in your App?

Since you have configured a maximum limit of 15,000 records in your application's data store, if you clear the data store and perform a full synchronization, would the number of records exceed the 15,000 limit? In other words, once the datastore reaches the 15,000 record limit, what would be the earliest date of the records that were synchronized and stored?

[5d]

@thisisabhash @5d Also I want to know, for our users in India Timezone, data is populating very fast but in US timezone data population is taking so much time.

Hi @amruth-movano , There are many factors that could influence the data synchronization speed.

1. In which region or geographic location is your application deployed?
2. What is the networking environment like for the devices you are using for testing? Are they connected to a local network, the cellular internet, or behind a VPN?

[amruth-movano]

Hi @amruth-movano ,

What is the typical or average volume/amount of data that is stored in your App?

Since you have configured a maximum limit of 15,000 records in your application's data store, if you clear the data store and perform a full synchronization, would the number of records exceed the 15,000 limit? In other words, once the datastore reaches the 15,000 record limit, what would be the earliest date of the records that were synchronized and stored?

Yes we have set max limit to 15000. So Ideally when we login it should store 15000 records from cloud to local cache. And we should get the data but in the current behaviour for some region we are not getting today's day data also immediately. Sometimes it is taking half an hour to load the data.

Yes we are able to see more than 15000 records locally when internet is off.

[amruth-movano]

Hi @5d, I tried updating my max sync records to 50000 and I am able to get the data. But on datastore.fetch call it's not returning anything, it is only showing the data when max sync records is done.

So fetching is not happening with datastore directly, it is happening with data which is synced locally i.e. from 50000 records. But we want to fetch the data from datastore directly.

Also while syncing datastore is syncing from oldest data first, is there any way to add sorting on how should it sync i.e. I want datstore should sync the latest data first

[amruth-movano]

Hi @5d @thisisabhash Can we connect if possible to discuss the exact scenarios?

[amruth-movano]

@5d @thisisabhash

Current Scenario - I set syncMaxRecodrs to 15000. It is only showing 15000 records from the first record for that users. i.e. if my first row time in db is 01-01-2024 then it is syncing first 15000 data from that date.

So, now issue is I am not getting my latest data though it is there in DB. The dataStore.query function from amplify datastore not returning anything but for older dates I am getting data from datastore.

[5d]

Hi @amruth-movano,

So fetching is not happening with datastore directly, it is happening with data which is synced locally i.e. from 50000 records. But we want to fetch the data from datastore directly.

The DataStore.query method queries the local SQL database and is designed to function even when the device is offline. Are you suggesting that you want to retrieve the data directly from DynamoDB?

Also while syncing datastore is syncing from oldest data first, is there any way to add sorting on how should it sync i.e. I want datstore should sync the latest data first

Currently, because the default database table in DynamoDB uses a unique identifier as the partition key, it is difficult to sync the datastore with the most recent records in order. Does your app need offline functionality? If not, you might want to consider using the API plugin directly. To perform mutations with the API plugin, you will need to disable conflict resolution.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.