search

tmaiaroto / discfg

A distributed, serverless, configuration tool using AWS services
Apache License 2.0
75 stars 4 forks source link

Stream to Kinesis #2

Open tmaiaroto opened 8 years ago

tmaiaroto commented 8 years ago

discfg has no long polling for updates like etcd. This is because Lambda is meant to be a short running thing. So it's not suited to long HTTP polling.

However. We can push out changes to AWS Kinesis. Then anything that's listening to that stream can do what it needs. So in a roundabout way, we can get the same functionality (perhaps slower though).

The really cool thing is that we can actually do more once we stream key changes (and other commands) to a Kinesis stream. We could be aggregating events for example. There can be other Lambdas reading from the stream at set message intervals, etc. There's a whole ton of possibilities. It might even change how people use discfg, which is ok too.

I'm thinking something like

./discfg set mycfg /somekey '{"some": "json"}' --stream

A stream might first need to be configured, otherwise (or in addition to) it could just be something like --stream=nameOfStream

This way, not everything needs to end up in the Kinesis stream. Which is also quite nice.

tmaiaroto commented 8 years ago

It might be nice to simply stream events in this manner too. We'll be 90% of the way there. See consul for some inspiration: https://www.consul.io/docs/commands/event.html

There's a lot of things I like about consul, but I'm beginning to wonder if some of those features are getting out of scope for discfg.