implement new aws sdk

[endepointe]

Kinesis sending to an S3 bucket works, great.

The data is still inconvenient to retrieve making it essentially unusable to the user. Whether this is a setup issue with S3, or something else, starting a rdms in the same way that the proof-of-concept S3 service was used may solve the immediate issue of making the data usable by the user.

[endepointe]

After reading the following whitepaper:

https://d0.awsstatic.com/whitepapers/whitepaper-streaming-data-solutions-on-aws-with-amazon-kinesis.pdf

I am going to implement the suggested solution to use KDF. The goal is to send the datastream to Elasticsearch and then see if a custom HTTP endpoint can be created to give the user more options.

Resources:

• https://docs.rs/aws-sdk-firehose/1.13.0/aws_sdk_firehose/
• https://aws.amazon.com/kinesis/data-firehose/
• https://aws.amazon.com/kinesis/data-firehose/pricing/?nc=sn&loc=3
• https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html
• https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html
• https://aws.amazon.com/what-is/elasticsearch/
• https://aws.amazon.com/opensearch-service/

Complete list of rust sdks:

• https://awslabs.github.io/aws-sdk-rust/

[endepointe]

Firehose is.... a firehose.

Dynamodb seems to be the best option until multiple destinations are made available. This should not be too difficult since trait impl allows each destination sdk to be implemented.

Creating a well-organized config file will make automation a breeze.

[endepointe]

adding opensearch is the next goal. Once done, then the option to send to multiple destination can become available.