New Construct: aws-lambda-redshift

[salunkhesl]

Use Case

When you have Redshift Cluster provisioned in Private Subnet, you cannot load the data from S3 directly.

Proposed Solution

I propose to use Lambda Function provisioned inside private subnet to load data from S3 Bucket using S3 Event mechanism.

Please find attached Architecture diagram as well as proposed Lambda Function package.

[biffgaut]

I believe this can be accomplished with current with the current aws-s3-lambda construct. When you specify lambdaFunctionProps for the s3-lambda construct, specify a vpc:

import { S3ToLambdaProps, S3ToLambda } from '@aws-solutions-constructs/aws-s3-lambda';

const existingVpc = Vpc.fromVpcAttributes( /* enough info to find VPC */);

new S3ToLambda(this, 'test-s3-lambda', {
    lambdaFunctionProps: {
        code: lambda.Code.fromAsset(`${__dirname}/lambda`),
        runtime: lambda.Runtime.NODEJS_12_X,
        handler: 'index.handler',
        vpc: existingVpc
    },
});

(did this off the top of my head, please excuse syntax issues) This will allow your Lambda function to access resources in the VPC.

[salunkhesl]

Not sure I understand your comment. The solution I am proposing is end-to-end to allow users to load s3 data into Redshift Cluster provisioned in private subnet. Are you saying, we have this lambda construct already? I don't see that in the list.

[hnishar]

salunkhesl@ Are you proposing the constructs will provision the Redshift resources/permissions in addition to lambda ? If so, we don't have aws-lambda-redshift construct in the library. We could then plug that with the existing aws-kinesisfirehose-s3 and aws-s3-lambda constructs to create the expected architecture you proposed.

[salunkhesl]

salunkhesl@ Are you proposing the constructs will provision the Redshift resources/permissions in addition to lambda ? If so, we don't have aws-lambda-redshift construct in the library. We could then plug that with the existing aws-kinesisfirehose-s3 and aws-s3-lambda constructs to create the expected architecture you proposed.

Yes, I intend to provide E2E solution to load into RD cluster in Private subnet. We could use existing constructs or I will simply create CFN including all the components.

[biffgaut]

An end to end solution built on other constructs or including CFN doesn't sound like a Solutions Construct. Our goal would be to publish small reusable blocks that you could use to create the end to end solution in the CDK.

Right now we have constructs to cover all of the use case except a Lambda function accessing a Redshift database - that's what Hitendra was recommending.

Check out our Design Guidelines for what makes a good Solutions Construct.

[salunkhesl]

Sure, I understand now. So, it would be a aws-lambda-redshift construct to accept RS Clusters parameters and S3 URL. I have already got the Lambda function which does the job. However, it needs to be modeled to follow the Solution Constructs standards.

[biffgaut]

Close - it would define a new Lambda function or accept an existing Lambda function, there would be no S3 aspect to it at all. Data would be read from S3 with an aws-s3-lambda pattern or perhaps an aws-lambda-s3 pattern, depending on what is triggering that activity.

This will be our first Amazon Redshift pattern - you will notice there is no discussion of common interface elements in DESIGN_GUIDELINES.md - so it will be defining how patterns interact with Amazon Redshift, which is a pretty big deal (how we interact with Lambda is pretty well defined at this point). As a first step, please write the README.md document that defines how the construct will work for us to review and collaborate on.

[biffgaut]

Also - we don't include any business logic in constructs, so the code for the Lambda function will not be included.

[salunkhesl]

Sure, I will put together README doc and share it with you for review.

[salunkhesl]

@hnishar , I wish to progress this however, I am not a JAVA expert to write this construct. Will I get any support for this development?

[biffgaut]

Solutions Constructs are written in Javascript, not Java - does that change your concerns?

[salunkhesl]

Am not a Java or javascript expert. Never worked with these languages.

[biffgaut]

If you've never used Typescript, it would be very difficult to write a new Solutions Construct.