search

bluehalo / yadda

Yet Another Docker Deployment Application
MIT License
4 stars 1 forks source link

Cron-lambda should use query instead of scan #7

Open jyoung-asymmetrik opened 7 years ago

jyoung-asymmetrik commented 7 years ago

I believe the scan in dynamodb will always scan the entire index, thus consuming the same amount of read regardless of the filter. The filter only cuts down on how many items are returned. http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScanGuidelines.html

bodnarbm commented 7 years ago

Unless DynamoDB has changed recently, query requires providing a specific partition key (which in the current case is the app name). Under the current structure, to do a query instead of a scan, would probably require a lambda per appName or a single lambda with knowledge of all the currently deployed apps and executing a query per appname.

The other idea I had before that could be implemented is to break out the current table into two. Keep the deployment history table to enable rollbacks, but also create a separate time that manages only the currently deployed/active jobs. That table should remain rather small, scaling up only with the number of active jobs instead of with each deployment.

jyoung-asymmetrik commented 7 years ago

Hmm I see. I was thinking perhaps we could use the active flag as a partition key, but I'm not familiar enough with DynamoDB yet to be able to determine if that is possible. I'll take a look when I get a chance. Thanks for the information.

jyoung-asymmetrik commented 7 years ago

Looks like you could use active and deployment time as a composite partition key. That would basically separate the table and should allow you to query with just active = 1. The appName shouldn't really be important for running tasks I think.