sam-goodwin / punchcard

Type-safe AWS infrastructure.
Apache License 2.0
507 stars 20 forks source link

Generalize the Enumerable services such as SQS, SNS, Kinesis, Firehose, S3 and Glue Tables and creates intuitive mappings between them #22

Closed sam-goodwin closed 5 years ago

sam-goodwin commented 5 years ago

This change generalizes the Enumerable services such as SQS, SNS, Kinesis, Firehose, S3 and Glue Tables and creates intuitive mappings between them.

Closes #1

Example (see: https://github.com/sam-goodwin/punchcard/blob/to-glue/examples/lib/stream-processing.ts)

/**
 * Persist Kinesis Stream data as a tome-series Glue Table.
 * 
 * Kinesis Stream -> Firehose Delivery Stream -> S3 (staging) -> Lambda -> S3 (partitioned by `year`, `month`, `day`, `hour` and `minute`)
 *                                                                      -> Glue Catalog
 */
const database = new glue.Database(stack, 'Database', {
  databaseName: 'my_database'
});
stream
  .toS3(stack, 'ToS3').enumerable()
  .toGlueTable(stack, 'ToGlue', {
    database,
    tableName: 'my_table',
    columns: stream.type.shape,
    partition: {
      // Glue Table partition keys: minutely using the timestamp field
      keys: {
        year: integer(),
        month: integer(),
        day: integer(),
        hour: integer(),
        minute: integer()
      },
      get: record => ({
        // define the mapping of a record to its Glue Table partition keys
        year: record.timestamp.getUTCFullYear(),
        month: record.timestamp.getUTCMonth(),
        day: record.timestamp.getUTCDate(),
        hour: record.timestamp.getUTCHours(),
        minute: record.timestamp.getUTCMinutes(),
      })
    }
  });