This change adds a DataLake Stack that persists all observed Domain Events, Issued Commands and Store state changes in a Bounded Context. The data is collected into a single Kinesis Stream and processed by an Apache Flink application running on an Kinesis Analytics managed Flink cluster. This application consumes from the stream, partitions the data by type and time and stores the data in S3 as encrypted JSON and Parquet files. These partitions are then updated in the corresponding AWS Glue Tables so that they can be queried in Athena, Spark and Hadoop (or any other Hive-compatible consumer). Data can also be configured to be loaded into an AWS Timestream instance to enable fast time-stream analysis.
TODO:
[ ] Implement AWS Glue and S3 Sink in stochastic-flink
[ ] Implement AWS Timestream Sink in `stochastic-flink
[ ] Enhance DataLake construct, making it so each table can be configured individually - use mapped types to map the BoundedContext to DataLakeProps.
Closes #84 Closes #95
This change adds a
DataLake
Stack that persists all observed Domain Events, Issued Commands and Store state changes in a Bounded Context. The data is collected into a single Kinesis Stream and processed by an Apache Flink application running on an Kinesis Analytics managed Flink cluster. This application consumes from the stream, partitions the data by type and time and stores the data in S3 as encrypted JSON and Parquet files. These partitions are then updated in the corresponding AWS Glue Tables so that they can be queried in Athena, Spark and Hadoop (or any other Hive-compatible consumer). Data can also be configured to be loaded into an AWS Timestream instance to enable fast time-stream analysis.TODO:
stochastic-flink
BoundedContext
toDataLakeProps
.