Dilip-Solanki-Logistic-Infotech
commented
2 years ago @iartemiev Any way to solve this issue?
Closed Dilip-Solanki-Logistic-Infotech closed 2 years ago
Dilip-Solanki-Logistic-Infotech
commented
2 years ago @iartemiev Any way to solve this issue?
PeteDuncanson
commented
2 years ago @Dilip-Solanki-Logistic-Infotech wow, I suspect your sync is scanning the DynamoDB table which has to be done 1Mb at time, with 1.4Mb thats a lot of info. Take a look at switching to get it to use a Query instead https://docs.amplify.aws/lib/datastore/sync/q/platform/js/#advanced-use-case---query-instead-of-scan
Can't say I've used that myself but its meant to help to get you out of a hole like the one you are in.
Cheers
Pete
chrisbonifacio
commented
2 years ago Hi @Dilip-Solanki-Logistic-Infotech 👋 I second @PeteDuncanson's recommendation. That is a lot of data, definitely want to narrow that down to a query and see if you can index on something and then filter by the timestamp.
You can try by adjusting this example from the docs Pete linked above
DataStore.configure({
syncExpressions: [
syncExpression(User, () => {
const lastName = await getLastNameForSync();
return user => user.lastName('eq', lastName).createdAt('gt', '2020-10-10')
})
]
});I need is a record of the last 8 hours. So, No other field to be there like, lastName.
PeteDuncanson
commented
2 years ago @Dilip-Solanki-Logistic-Infotech your other option is to just use GraphQL direct for that data but then you would lose the offline goodness if you go that route. Welcome to my world :)
chrisbonifacio
commented
2 years ago It might be worth a try but I would expect GraphQL to still be slow if the query is still performing a scan rather a query on the DDB table, because it'd still be hitting all items in the table. Not including a query expression will result in the same operation type (Scan vs Query) on the table.
For example, here's the relevant part of the vtl template for a CLI-generated list query:
#if( !$util.isNull($ctx.stash.modelQueryExpression) && !$util.isNullOrEmpty($ctx.stash.modelQueryExpression.expression) )
$util.qr($ListRequest.put("operation", "Query")) // < -- if query expression is not NULL or Empty (ex. allPostsByAuthor({ author: "Chris" })
$util.qr($ListRequest.put("query", $ctx.stash.modelQueryExpression))
#if( !$util.isNull($ctx.args.sortDirection) && $ctx.args.sortDirection == "DESC" )
#set( $ListRequest.scanIndexForward = false )
#else
#set( $ListRequest.scanIndexForward = true )
#end
#else
$util.qr($ListRequest.put("operation", "Scan")) // < -- otherwise, performs Scan by default (ex. allPosts())
#end@Dilip-Solanki-Logistic-Infotech Perhaps another route might be to paginate the results instead? You can use DataStore.query instead of the syncExpression like so:
https://docs.amplify.aws/lib/datastore/data-access/q/platform/js/#predicates
const posts = await DataStore.query(OrderEvent, event => event.timestamp("ge", startDate) , {
page: 0,
limit: 100
});Otherwise, if you'd prefer to use the syncExpression approach in the DataStore configuration, one thing you might be able to do to work some kind of GSI into the schema is set a fuzzy date so that the query doesn't have to hit every item in the table. This might be easier to work in so your logic doesn't need to change as much.
example:
DataStore.configure({
syncExpressions: [
syncExpression(OrderEvent, () => {
return event => event.createdAtFuzzy('eq', '2022-01-12') // or today's date, or today .or() yesterday's dates and then filter further from here
})
]
});you can add this field to your model to facilitate this
createdAtFuzzy: String!
@index(name: "byCreatedAtFuzzy", queryField: "orderEventByCreatedAtFuzzy")Let me know if this helps. Otherwise, I can check with the DataStore team to get some more feedback for you. 🙏
PeteDuncanson
commented
2 years ago @chrisbonifacio I don't think he can use DataStore at all for that amount of data, even paging it within DataStore means it still needs to have it locally so still the same issue, pulling a tonne of data down up front right? The paging is just paging through local data.
GraphQL hit with a GSI would be a good way to in this case, as usual, work out your access pattern need up front and build a GSI or query to match it efficiently. Ha! Listen to me, 18 months of burnt fingers at least I've learnt something :)
chrisbonifacio
commented
2 years ago @PeteDuncanson ah, that's a good point, it won't apply to the sync. The syncExpression with a GSI might be the best approach in a DataStore context then because the expression would apply to base and delta syncs and incoming subs. Otherwise, if that doesn't help to speed up the sync then GQL and GSI would be the way to go.
Dilip-Solanki-Logistic-Infotech
commented
2 years ago @chrisbonifacio @iartemiev Can you please give a simple example as per my requirement? I just need 8 hours records with datastore and syncExpression. My whole project depends on the datastore only.
chrisbonifacio
commented
2 years ago @Dilip-Solanki-Logistic-Infotech I would suggest trying what I mentioned in my previous comment: https://github.com/aws-amplify/amplify-js/issues/9471#issuecomment-1017905677
OrderEvent {
# ...other fields
createdAtFuzzy: String! @index(name: "byCreatedAtFuzzy", queryField: "orderEventByCreatedAtFuzzy")
}DataStore.configure({
syncExpressions: [
syncExpression(OrderEvent, () => {
return event => event
.createdAtFuzzy('eq', '2022-02-2') // or today's date, or today .or() yesterday's dates and then filter further from here
.timestamp('ge', startDate)
})
]
});If you can try this, let us know if you experience better response time.
chrisbonifacio
commented
2 years ago Hi 👋 Closing this as we have not heard back from you. If you are still experiencing this issue and in need of assistance, please feel free to comment and provide us with any information previously requested by our team members so we can re-open this issue and be better able to assist you. Thank you!
Dilip-Solanki-Logistic-Infotech
commented
2 years ago @chrisbonifacio Thanks. I will try to your suggestion.
github-actions[bot]
commented
1 year ago This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.
Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels or Discussions for those types of questions.
Before opening, please confirm:
JavaScript Framework
React
Amplify APIs
DataStore
Amplify Categories
No response
Environment information
Describe the bug
When logged in the started the initial syncing on my app. I used the syncExpression for some models. But one model (OrderEvent) takes more time to sync. Because it is synced sometimes with one or two records only. I put some screenshots for that.
Sometimes takes 5 to 10 minutes to complete sync.
Total Items in OrderEvent: 1,436,618
Expected behavior
I want to solve the issue. But, I don't why it happens only in a particular model.
Reproduction steps
Add the
syncExpressionwith the condition. Open the Inspect bar. Click theNetworktab. See thesyncOrderEvents.Code Snippet
Log output
aws-exports.js
No response
Manual configuration
No response
Additional configuration
No response
Mobile Device
No response
Mobile Operating System
No response
Mobile Browser
No response
Mobile Browser Version
No response
Additional information and screenshots