RFC: Event persistence

[camjackson]

Soon we'll start to switch things over from a relational data store to an event sourcing model. One of the biggest questions to answer is what exactly we're going to use for our event database.

Some criteria, off the top of my head:

• Suitable for storing a schema-less event stream
• Performant both in terms of throughput and capacity
• Ideally supports some sort of competing consumers pattern
• API that's easy for developers to use (or easy to abstract so that it's easy)
• Easy to manage from an operations point of view
• 'Scales down' so that orgs on a budget can run it on a single, small node
• Easy to install locally (or at least automatable install)

Some contenders:

Event Store:

• + Designed for this kind of use case, with streams, projections, competing consumers built in.
• + Docker image that's easy to run locally

CouchDB:

• ???

DynamoDB

• + Managed by AWS - very little ops work to get started with it
• + Managed by AWS - very little maintenance for orgs without technical staff
• + Only pay for actual usage, so it can be super cheap
• - Locked in to AWS big time.

MongoDB

• ???

Relational data store e.g. postgres

• + Well-known technology
• - Not really designed with this use case in mind? (Does that matter?)

There's probably a bunch more research that needs to be done before we can make a call, but right now I suppose the biggest question is to what degree we want to be locked in to AWS. On the one hand it gives us a huge advantage in terms of up front infrastructure work now, and maintenance cost later. But on the other hand it may become quite difficult to move away from if we wanted to. Locking our core data store in to AWS is a bigger deal than using S3 for example.

Please add your corrections and additions to the above!

[camjackson]

We're using kinesis for now, with lambdas to forward events to services over HTTP.