PeteDuncanson
commented
3 years ago I've been thinking about this one as its an area in our product development we are now getting to. I'd like to warm this up with a few options that hopefully might lead to something. Like a lot of others I'd just assumed that this would be trivial, a fault of my not understanding document databases well enough I guess. Its a bit of a trap to fall into when so far down the line of developing, try telling your client/manager that you need 5 days to get out the total records for something...its not a good look.
One issue with this whole thread seems to be the assumption that whatever the solution it has to cope with zillions of records when in fact a lot of apps might only be counting in the 100's or 1000's in some tables so I don't think there needs to be a one size fits all solution. I get that some of those solutions could cause some big issues if mis-used but with some knowledge of any constraints or suggested working parameters that should be safely navigated.
I'll start off with the smallest (and some might say naive) solution first and build up to the bigger ones. Remember, there isn't a one size fits all solution. My focus is on trying to make it easy to add this functionality with minimal hassle from the "developer building out the schema" point of view.
Using a scan to count records
If you don't have many records (< 1000 or so) you might be able to get away with using the scannedCount field as described here https://www.riccardosirigu.com/blog/aws-amplify-dynamodb-total-count-graphql-queries/ although it can "top out" when it hits its 1MB result set limit and so report a false total. Ideally it would be nice to know if there are more that could have been returned or if that is the total amount i.e. was there another page of results possible? That would at least give you the option of saying "record 1 of at least 246+" or something if we can't get an exact amount back.
Another option here is to do a full paged scan if I opt into it so I can get a "true total". This would loop through all the pages of results for a scan and return the total. I keep hearing that scans are expensive but again if I've only got a relatively small number of records that might be a hit I'm happy taking for now and just swallow the hit and push off doing it "properly for epic scale" in the future when it becomes a happy problem (ie we've grown so much we need to change it).
I can see this being a new directive that could be added to a model, something like:
type Posts @model {
title: String!
content: String!
author: String!
published: Boolean!
totalPosts: Int! @scannedCount
}This new @scannedCount directive would automatically do a scans of all the records in the Post table that could page through all the results until it finishes (thus masking from us the issues with the 1MB returned data limit). Used as shown above it would just give the total of all records for that table.
Now if we ignore the obvious performance traps that this directive sets up for us (we will discus those next) I think we can all agree that from a users point of view that looks like a simple and clean way of getting what we want, its a super quick way to get a total value out, problem solved from the users point of view initially. For fast prototyping or "good enough" features that might do me.
So now on to the problems this will cause or that need solving. Luckily now we have a directive we can add to it with some attributes to give us some finer grained control or indeed we could create multiple directives to give us the following functionality.
A quick note on Lambda functions, we are going to need some generating, would it be worth while if these are generated and stored in the /backoffice/functions/ folder as user generated functions are. That way should a user want to extend the function with additional logic they can. The functions this directive might generate should be all nicely stubbed out with pointers on where and how to add additional logic and would need pushing to deploy like any other user generated function. This directive shouldn't hide all the magic in the cloud and VTL templates, it should do the minimum to make the common case go a way and then give the user an extension point to add additional logic as their needs demand. This might mean that removing the directive from the schema won't delete the resources from the cloud automagically if you've edited them, I'd be happy with a warning about "Some functions appear to have been edited so you will need to manually delete the following or (A) delete all anyway".
Too many paged scans
If left unchecked this could get out of hand with the number of scanned pages growing over time so as a fail safe we should have a default number of pages that this will loop over before returning. The default can be agreed later but if the user wants to opt-in to setting it then they could in the directive like so:
totalPosts: Int! @scannedCount( maxPageDepth: 5 )The maxPageDepth would be an optional param to set how many pages you are happy to do to get that total to stop it all grinding to a halt in the future (but hey we opted into this solution, its not ideal its pragmatic and we know the trade offs).
The trouble with the above is the same as the scannedCount solution from the blog post, it doesn't tell you if it reached the end of the pages or not, so you've now way of knowing if the reported number of results is the total amount or just the amount it got before hitting maxPageDepth. But it might be good enough for the right sized app and it sure is simple to do from a users point of view. As long as the limitations are documented they users should be able to make an informed choice.
Additionally maybe a trigger in CloudWatch could be created as part of using this directive to ping a notification when a field is taking too long to rescan? A nice bit of Developer Experience and another way to introduce a new feature to those of us still learning our way around what AWS has to offer.
Total results for filters, sort keys or conditions.
I want something other that the total count, I want a count of some filter or other. This ones a bit trickier but could we allow filters in the directive?
totalPosts: Int! @scannedCount
totalPublishedPosts: Int! @scannedCount( filter: "published=true") #something like that anywayThat would work but its not dynamic, I can't pass in a filter at runtime but it would be good for various counters/stats that could be handy to have. I wonder if we could generate a GraphQL query that would allow you to pass a filter through to this directive/resolver (the lines become blurred as to which this now is at this point, guessing its a resolver under the hood but then I don't know enough to know what a resolver does under the hood as yet so my terms might be wrong here, feel free to enlighten me).
Sort keys could be passed through too maybe to give you a total for a subset in a similar manner, its not my strong point so this part of the ideas is hazy on purpose but here to get you thinking.
Another issue that @houmark raises repeatedly is that the number of results that match a filter can only be worked out if we scan all the records and also it doesn't work if we have a limit. This @scannedCount would do both I think in its own brute force way.
Caching.
Do we really want or need to look this total up every time? If its a small number of records then why not, it wouldn't be all that slow nor costly (again we are talking about relatively small number of records here not eBay products or Uber rides size apps) and we could cache it on the frontend if needed.
But if you want to opt in to caching we could do it one of two ways that could be set via the directive at the expense of another table and a function.
You could do this with a simple boolean or enum (to allow for more options in the future):
totalPosts: Int! @scannedCount( cached: true )I'm not 100% sure of the internal here so I'm going to propose what I think it could do with my understanding of the options, there might be other better ways but this should give the gist. We need to store the value somewhere and then pull it back out.
One way would be to do this under the hood:
- Spin up a new dynamodb table to store all the totals in, one row per requested "scannedCount" directive in the schema. If another
@scannedCountdirective has already created this table we can just re-use it? Along with storing the total would be also store some other fields like model name, target field, createdAt and updatedAt. - Then we automatically generate a Lambda function that would invalid or recalculate the value for the cache. There are a few ways to do this that I'll discus below.
Recalculate the cached value for any CRUD operation
This version uses a Lambda trigger function using DynamoDB Streams (one of the options when creating a function via the CLI so a known thing) that hooks into this models table updates and will rescan the table whenever a record is added/removed. This is a bit brute force but it would work.
The trouble with his method is it could cause needless thrashing of the tables when doing an import, we don't need that number generated on every entry if we are adding 100's or 1000's of records. I might be that you can live with the hit if the numbers are small but an option to disable the caching while doing an import but still trigger it to do a final count at the end would be good. I sadly don't know enough of how to do that nicely. You could add a "disabled" boolean attribute to the directive maybe but that would need another push to get it up to disable it and then how to get it to recalculate after enabling it again without just "touching" an existing record afterwards? That might be enough?
Invalid the cache on any CRUD operation, recalc when requested if needed
totalPosts: Int! @scannedCount( invalidOnCRUD: true ) # open to suggestions on any of these names btwAnother option is similar to the above except we don't do the rescan when the record changes we just null the value in the cache table. Then our resolver that gets the total for that field will do the rescan for us if the value is null. Depending on the data model and workflow of the app this might be a better fit.
My only worry here is that we separate out the two bits of cache work to two separate areas, the function to clear the cache and the resolver to build the cache.
Generate the totals on a recurring timespan
An alternative if we are ok with the totals potentially not being 100% accurate at any given time is to have a timed Function do the rescan repeatedly after a set timespan. Again this could be set via the directive:
totalPosts: Int! @scannedCount( refreshCountInSeconds: 300 ) # regenerate every 5 minutesI don't know the best practise for passing timespans around so that is up for debate but hopefully you get the idea.
How to get the value from the cache into our model?
How we get that cached value out into that model though I'm hazy on, I guess we just have a resolver generated which would go get the value from the "cache table" but better minds than mine will know how best to do that.
Rather than having these fields on the model themselves do we need a PostTotals model generated instead that can store all these top line numbers on and possibly pull these from the cache table via a resolver or when generated by amplify codegen etc? In that case does the directive move up to the whole table rather than the field level?
type Posts
@model
@scannedCount( fieldName: "totalPosts" ) {
title: String!
content: String!
author: String!
published: Boolean!
}Running totals for larger record sets
What if we out grow the @scannedCount directive above, what options do we have? Scanning is too slow due to the number of records we've got. This might be where a @runningTotal directive might be the answer.
It has been suggested above that you can keep a rolling count of various totals by hooking into the "DynamoDB stream" which will fire an event every time you add/update/delete a record in DynamoDB. We could then use these to increment or decrease a totals counter.
This solution though has some instant problems:
- Its not all that user friendly to tell developers they have to do all this additional work for something that should in a lot of minds be "baked in" - that is not to say that Amazon should be expected to do all the work and I'm sure that it is doable in a morning or so to wire something together but it just kills productivity, adds doubt to "have we made the right choice here?" and Amplify is meant to be about speeding up the gluing together of these services to let you focus on the "building an amazing app" rather than becoming an expert in yet more Amazon services and DevOps from day one. Just testing that setup can take a while...and they how much testing do you need before you can truly "trust it" and sleep at night I wonder? Better I think that Amplify make some of pain go away for the obvious use cases and then give users with edge cases place to start from.
- In works for the top line numbers but not so much for edge cases - again these could be handled by the user but the boiler plate for this could be put in place and the functions for it stored in the solution so they can be edited and expanded on.
I'd like to see a directive that could wire all this up for me from the schema, this would spin up the functions needed (stored locally for editing) and store the values in a separate table similar to the cache table mentioned earlier if not the same table but different records so we don't clash? Lambda trigger functions hooked into DynamoDB streams would do the adding/subtracting. As these are built to run all the time we won't have the same potential scaling/performance problems we had with the @scannedCount directive.
type Posts
@model
@runningTotal( fieldName: "totalPosts" ) {
title: String!
content: String!
author: String!
published: Boolean!
}Now what about that idea to have multiple counts on a table? Could we do this:
type Posts
@model
@runningTotal( fieldName: "totalPosts" )
@runningTotal( fieldName: "totalPublishedPosts", filter: "published=true" ) {
title: String!
content: String!
author: String!
published: Boolean!
}I don't see why not. Need more power? They you should be able to go edit the function code or copy it and now you've got a great starting point to adding in a few more lines to get you more counters as needed. Easy of development continues and I think we are still using all the tech that Amplify already has and uses so not a massive stretch of knowledge or comfort area?
I want more POWER!
Getting into wish list land with this last one, suggest the above gets done before this one gets looked at but...
Final option is something similar to what @RossWilliams suggested above that of "going large" and streaming the table data from dynamo into a Relational database so you can do full on SQL magic on that. This one currently is beyond my pay grade, I laughed when I first read Ross' comment and it sounded crazy complicated but after a bit of reading I can see how it all might stitch together...but again...should someone have to do all that to get some stats or counters out of their data? The answer might simply be "you are using the wrong database in the first place!" and I should go with a relational database from the off...but that isn't how Amplify is marketed or sold to me!
The basic idea is that by hooking into a DynamoDB Stream we can push record changes to a relational database and then from there we can pull out whatever data SQL will allow us to do. Sounds excessive but if its what we've got then lets use it. Could another directive handle this one?
type Posts
@model
@streamToSQL { # Guess this would be RedShift?
title: String!
content: String!
author: String!
published: Boolean!
}This would push any CRUD operations into a relational database, I guess you would need to add this database first as a resource via the CLI amplify add sql. It would create a table name the same as the model (could allow you to over-write this in the directive).
What to do about relations and @connections though. In theory it could include those and I guess that would be the dream.
This does raise the idea that do we need a @sql resolver similar to the @http resolver so we could quickly get the results of a sql query into our GraphQL? I'll leave that one for another issue I think ;)
What about auth?
@houmark raises the tricky point about authentication earlier and to be honest I don't know. I'm not all that sure if this is an issue with passing @auth to elastic search or just within the api. I'm hoping its an elastic search thing which would mean that the above solutions should work with the possible exception of the relational database idea, I can see that needing a lot more thought.
It should be possible to apply @auth to the model or the individual fields like normal which should cover it for the @scannedCount and @runningCount directives. Alternatively if we use a separate table like PostTotals a discussed above then @auth could be applied maybe to that? That should re-use the logic already in place I think?
Summing up
Sorry this is so long, it wasn't meant to be, its a tough problem and I've been giving it a lot of thought as I hope you can tell. I'm hampered though by only knowing some of what is needed to know the best route but hopefully this adds to the conversation and brings a users point of view to how I'd personally like it to roughly work and what constraints I might be happy living with to get some of the functionality I and others say we need.
I've tried really hard to leverage what Amplify already has or uses so that it shouldn't be a massive stretch of the imagination or skillset to get it to do what it needs to.
Lets keep the conversation going, what have the Amplify devs come up with in the time this has been open, where might it go and how can we get there (and ideally an idea of when)?
RossWilliams
ilyagru
duwerq
andre-scheffer
undefobj
simpson
snahammond
nubpro
loganpowell
xitanggg
chrisyeung1121
pagameba
GeorgeBellTMH
SalahAdDin
sacrampton
houmark
ataibarkai
hackrx
zkwentz
kgoyal98
chris-lee-code
matart15
warlord987
biller-aivy
Is your feature request related to a problem? Please describe. I think it's way past due that Amplify supports total counts in GraphQL queries. Other GraphQL based platforms has this built in by simply adding
totalCount(or similar) in the query and no matter the limit, they'll get back the total count in addition to the (filtered) data.Describe the solution you'd like This should at least work for DynamoDB backed models and of course also for search based queries that pass by ElasticSearch.
Describe alternatives you've considered Making a Lambda function that is a field in each model using the
@functiondirective, but since we are both usinglistItemsandsearchItemswith filters added, the implementation is not simple as we have to reapply those filters on the lambda function for getting the correct count.Making custom resolvers seems like another not very fun route and not very scaleable or maintainable, and once again, this should be a "out of the box one liner" to have available as a developer. With either a Lambda or some other custom resolver I'm looking at hours or days of development.
Additional context This is a must have feature and there's not really any workaround for displaying total counts for systems with many items — at least that I know of. I read several bug reports, but none of them seems to have a simple solution. That it has not yet been developer by AWS is beyond my understanding, as pulling out counts is one of the most common things to do when developing web apps.