search

Clever / wag

sWAGger - Web API Generator
Apache License 2.0
77 stars 6 forks source link

Support limiting the number of items returned by DynamoDB scan operations #312

Closed prime-time closed 3 years ago

prime-time commented 3 years ago

This change exposes a new optional Limit parameter for generated scan methods which allows scanning up to Limit number of items instead of every item. The motivation is that I want to create a new API endpoint which retrieves all items from a DynamoDB table, however there are too many items to do a full table scan in a single request. With this change, I'll be able to implement paging using scan.

The Limit parameter introduced here works basically the same way as Limit does for queries. I literally copied that code ©️ 😄 . https://github.com/Clever/wag/blob/a9b710a2f10a0cedf85531b0b409113db8072fc4/server/gendb/interface.go.tmpl#L120-L121 https://github.com/Clever/wag/blob/a9b710a2f10a0cedf85531b0b409113db8072fc4/server/gendb/table.go.tmpl#L434-L438

For reference, this describes how Limit works in ScanPages: https://docs.aws.amazon.com/sdk-for-go/api/service/dynamodb/#ScanInput

I manually tested this change on an API endpoint that I'm working on, and it works as expected.

I also added in a comment mentioning why the scan tests are structured differently which came up in #311

I recommend viewing the changes by commit.

ghirsch1 commented 3 years ago

Is this a one-off/infrequent thing? If so, maybe better to just export the table to S3 and use that? https://aws.amazon.com/blogs/aws/new-export-amazon-dynamodb-table-data-to-data-lake-amazon-s3/

rgarcia commented 3 years ago

also re:

The motivation is that I want to create a new API endpoint which retrieves all items from a DynamoDB table, however there are too many items to do a full table scan in a single request. With this change, I'll be able to implement paging using scan.

generally speaking scans in the request path are not ideal. Querying an index is preferred. You might to share details of the use case in #eng-infra / #dynamodb slack to make sure you won't have a bad time™

prime-time commented 3 years ago

The use case for the new endpoint I'm working on is for a script and potentially others in the future, so this wasn't something I was planning on using in application code. I posted the full story in #dynamodb (apologies for the novel 📚 ).

prime-time commented 3 years ago

We had a follow up discussion about this via slack, and there were no objections so I'll go forward with merging this 😄