Evaluate Amazon Kinesis

[sopel]

This is the companion issue to #273 resp. the motivating use cases - Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale:

Amazon Kinesis can collect and process hundreds of terabytes of data per hour from hundreds of thousands of sources, allowing you to easily write applications that process information in real-time, from sources such as [...] financial information, [...] and operational logs and metering data.

[...] you can build real-time dashboards, capture exceptions and generate alerts, [...] and make other real-time business or operational decisions. You can also easily send data to a variety of other services such as Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, or Amazon Redshift. In a few clicks and a couple of lines of code, you can start building applications which respond to changes in your data stream in seconds, at any scale, while only paying for the resources you use.

• :information_source: it would be trivial to summarize the similarly appealing implications of this design for our use case as I did for #273, so I'll skip this - see Functionality for details

Not surprisingly, this new service is specifically positioned as a replacement for the most common OSS solution in this space, namely Apache Kafka for ingres/storage plus (soon also Apache) Storm for processing - a few (paraphrased) quotes from a related webinar:

• Do you want to accelerate your rate of ingesting without incurring operational burden/overhead?
• Do you currently use Kafka, or something similiar with (or without) Storm for performing continuous ingestion of data?
• Do you want to create the next-gen continuous real-time processing oos with latencies of a few seconds ...?

Similarly, one sample use cases from the financial services industry matches our requirements specifically:

• maintain real-time audit trail of every single market/exchange order
• ingest all market order data reliably, and build real-time auditing applications

It seems reasonable to evaluate whether it might be the proper solution for resilience and scale out for the messaging pipeline (see e.g. #269 and #272) in principle - there are currently two potentially major limitations for our current use case though:

• :exclamation: it is currently only available in us-east-1, thereby increasing latency (minor) and esp. cost for outbound streaming to applications/storage in eu-west-1
• :question: there are obviously not as many readily available integrations (e.g. for logstash), though these will supposedly be easy to write (the current Amazon Kinesis Client Library is Java only though)

[sopel]

Moved to Icebox due to aforementioned limitations and focus on #270 (as discussed in https://github.com/cityindex/logsearch-config/issues/59).

[sopel]

Closed as Won't Fix in favor of migrating to Kafka, see #273 and https://github.com/logsearch/logsearch-boshrelease/issues/73.