Closed sam-goodwin closed 5 years ago
This change generalizes the Enumerable services such as SQS, SNS, Kinesis, Firehose, S3 and Glue Tables and creates intuitive mappings between them.
Enumerable
Closes #1
Example (see: https://github.com/sam-goodwin/punchcard/blob/to-glue/examples/lib/stream-processing.ts)
/** * Persist Kinesis Stream data as a tome-series Glue Table. * * Kinesis Stream -> Firehose Delivery Stream -> S3 (staging) -> Lambda -> S3 (partitioned by `year`, `month`, `day`, `hour` and `minute`) * -> Glue Catalog */ const database = new glue.Database(stack, 'Database', { databaseName: 'my_database' }); stream .toS3(stack, 'ToS3').enumerable() .toGlueTable(stack, 'ToGlue', { database, tableName: 'my_table', columns: stream.type.shape, partition: { // Glue Table partition keys: minutely using the timestamp field keys: { year: integer(), month: integer(), day: integer(), hour: integer(), minute: integer() }, get: record => ({ // define the mapping of a record to its Glue Table partition keys year: record.timestamp.getUTCFullYear(), month: record.timestamp.getUTCMonth(), day: record.timestamp.getUTCDate(), hour: record.timestamp.getUTCHours(), minute: record.timestamp.getUTCMinutes(), }) } });
This change generalizes the
Enumerable
services such as SQS, SNS, Kinesis, Firehose, S3 and Glue Tables and creates intuitive mappings between them.Closes #1
Example (see: https://github.com/sam-goodwin/punchcard/blob/to-glue/examples/lib/stream-processing.ts)