search

mosuka / phalanx

Phalanx is a cloud-native distributed search engine that provides endpoints through gRPC and traditional RESTful API.
Apache License 2.0
359 stars 27 forks source link

Allow the DynamoDB metadata store to send changes made to DynamoDB as storage events. #83

Open mosuka opened 2 years ago

mosuka commented 2 years ago

Allow the DynamoDB metadata store to send changes made to DynamoDB as storage events.

Implement the following watch function.

https://github.com/mosuka/phalanx/blob/6ac57943b65dbedcf2504ba8459f3ffb7346bb6c/metastore/storage_dynamodb.go#L108-L126

It would be better if it could be implemented in the same way as the etcd metadata store.

https://github.com/mosuka/phalanx/blob/6ac57943b65dbedcf2504ba8459f3ffb7346bb6c/metastore/storage_etcd.go#L65

wolfeidau commented 2 years ago

@mosuka I don't think this can be achieved just using dynamodb as it is a little leaner on these sort of features than etcd.

DynamoDB could be paired with DynamoDB streams, lambda and a service which provides filtering of events like eventbridge, which also requires SQS to enable polling see project below.

Overall you end up with quite a few serverless components to do the same thing, each of which plays it's part.

It would be good to understand the volume of changes you would be monitoring, this kind of dictates how things will scale cost wise, and which services to use.

This https://github.com/spezam/eventbridge-cli is a good example of using eventbridge paired with SQS to enable selective monitoring for events.

mosuka commented 2 years ago

@wolfeidau Thanks for the comment. I will try to use this as a reference.

wolfeidau commented 2 years ago

@mosuka I found this recently, worth digging into how it works just to give you more context about how dynamodb streams can be used.

https://github.com/aws-samples/serverless-lifo-queue-demonstration