search

mbudm / fotopia-serverless

An API, storage and face detection for a photo archive web app using serverless framework
21 stars 5 forks source link

Indexes updates need fan in pattern #20

Closed mbudm closed 5 years ago

mbudm commented 5 years ago

The put indexes lambda is invoked by the streams lambda that is triggered by dynamodb creates, update and removes.

Occasionally two events occur simultaneously (within a few 100ms of each other) and they get the same existing state and write a new state. Not a big deal at small scale but will become a bigger issue (and before I'm ready to implement proper search indexes.

So the additon of an SQS resource should do the trick - force each put index to complete before the next one is tackled. And/or use a dynamodb table for faster get/put.

mbudm commented 5 years ago

From this article https://serverless.com/blog/aws-lambda-sqs-serverless-integration/

The plan would be to have a

reservedConcurrency: 1

At the initial small scale this would ensure there are no race conditions, in updating the indexes S3 object. Obviously this has scaling limits but that's ok for now. Or use a dynamodb for storage.

mbudm commented 5 years ago

Fixed - final tweaks in #24