Mapping Template APPSYNC_JS runtime: Array.prototype.reduce silently stops at 1000th element

[timheilman]

I've built a small CDK application to verify this behavior.

I would like to process over 1000 (very small) results from a DynamoDb query result set in an AppSync response mapping template. All of the results are delivered to the response mapping template, but calling reduce() on the resultset items array only processes the first 1000 results, silently returning the accumulator without processing the remaining results thereafter.

Why are we limited to 1000 items in the reduce() call? And if this is simply a shortcoming of the APPSYNC_JS runtime, why on earth wouldn't it be documented anywhere? This page claims that Array.prototype.reduce is supported, and says NOTHING about a limit of 1000 items. Can someone help me understand what is going on?

[timheilman]

I've verified that the same 1000 (or 1001)-element limit is in place for Array.prototype.map and with the for .. of syntax.

[timheilman]

It seems slice seems to not-mind parameters > 1000. This suggests the failing methods can be made to work again by manually batching to 1000-element chunks using slice, and manually passing the accumulator into the next reduce call.

[timheilman]

Array.prototype.filter also suffers from this limitation. I'm successfully working around it by using a loop-of-loops where no individual loop goes over the 1000 count, but it feels simply insane.

[naedx]

Hi @timheilman , I believe the limitation comes from "Iterations in a foreach loop in mapping templates" described here:

Iterations in a foreach loop in mapping templates Each supported Region: 1,000 Adjustable: No Maximum number of iterations in #foreach...#end loop in mapping templates

The doc references VTL resolvers specifically but I think it applies to both VTL and JS runtimes. I think the limits are imposed because AppSync resolvers are supposed to be quick to execute and return a response. Note that there's an execution time limit of 30 seconds among the list of limits, so while your workaround works at the moment you could still hit this limit.

I think you'll ultimately have to consider a different approach to processing this data. These might be worth considering:

• https://community.aws/content/2hlqAp86YvckSS2DrVvZ1qdArqF/async-lambda-and-appsync?lang=en
• https://dev.to/aws-builders/executing-long-running-tasks-with-appsync-1ikk
• https://aws.amazon.com/blogs/mobile/invoking-aws-step-functions-short-and-long-running-workflows-from-aws-appsync/

[timheilman]

Thank you so much for responding, @naedx ! I was starting to feel all alone in the world.

I had found other references to the 1000-iteration loop limit for the VTL runtime, but I had not found the quotas page that you quote. It’s very helpful. Thank you!

For anyone else finding this post, I am confirming that this is not just the VTL runtime but the JS runtime too, for map, reduce, filter, and for .. of.

I’m aware of both the 30-second time limit for AppSync endpoint execution and the 1 MiB data limit delivered to the response mapping template. My use case is nowhere near either of these limits. The workaround is still returning results instantaneously, and I’m in pre-production: the entire DynamoDB database is not yet 1 MiB.

Thanks also for the references to async invocation of lambdas and lambda step functions. They are good food for thought, however, I require no lambdas at all for this AppSync endpoint.

I’m simply using vertical partitioning. Using the loops-of-loops workaround does cause me anxiety, and I recognize I should find some better way to avoid 1000+ iteration loops, but I believe it’s the design of this vertical partitioning that I may need to change. The issue I’m encountering is that I have a hierarchy in the sortKey:

HK,SK1,(main entity attributes) HK,SK1#SUBENTITY,(subentity attributes) HK,SK1#SUBENTITY#SUBSUBENTITY,(sub-subentity attributes) HK,SK2,(main entity attributes) HK,SK2#SUBENTITY,(subentity attributes) HK,SK2#SUBENTITY#SUBSUBENTITY,(sub-subentity attributes)

I’d like to be able to retrieve all of SK1’s entity and (sub-)subentity data efficiently in a single query. I accomplish this with begins_with(sortkey, "SK1") These queries do not hit the 1000-record limit.

The issue is arising in my second use case: to be able to list all main-entity attributes for a particular hash key. Due to this particular vertical partitioning design, I have no choice but to return all records for that hash key, and discard any records with a sortkey containing #. I think DynamoDB’s filtering mechanism will not work on sort keys, so perhaps my solution will be to add a mainentity non-hash, non-sort boolean attribute on the main entities, then use the filter mechanism to reduce the number of results returned from dynamoDB. The 1 MiB limit still applies to the list prior to filtration, but at least this way I’ll get a nextToken to return if I hit that 1 MiB limit, rather than having a naive reduce silently discard the records returned beyond 1000.

It bothers me that I’ll be aliasing the “main entity” nature of my main entities — both that the sortkey does not contain “#” and that the mainentity boolean is set to true, since this introduces the potential for the alias to be out-of-sync: that a sortkey might contain “#” and have mainentity set to true, or vice-versa. I’ll have to weigh which is worse: aliasing the mainentity-ness of the record or using the loop-of-loops workaround.

In any case, I find it unprofessional that A) reduce and the other iteration constructs silently discard results beyond 1000; throwing an exception instead would have greatly reduced my time spent on this goose chase, and that B) this behavior isn’t documented explicitly for the JS runtime at all in the official AWS documentation. I hope this post can help some other poor soul scratching their head about this behavior.

[naedx]

I'm sure that a developer or two may have pulled their hair out with a silent failure here. I agree that an explicit exception would be useful (esp. in dev mode).

Would you be able to leave some feedback on the docs? I just noticed that there's a (feedback link)[https://docs-feedback.aws.amazon.com/feedback.jsp?hidden_service_name=AppSync&topic_url=https://docs.aws.amazon.com/appsync/latest/devguide/resolver-reference-overview-js.html] at the top, right of the docs pages. Perhaps some feedback there will help the AppSync team to improve the docs and perhaps handle these limits more explicity and transparently.