ashishdhingra
commented
2 days ago Reproducible using below code (TypeScript):
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as redshift from '@aws-cdk/aws-redshift-alpha';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
export class RedshiftCdkStack extends cdk.Stack {
constructor(
scope: Construct,
id: string,
props?: cdk.StackProps
) {
super(scope, id, props);
const vpc = new ec2.Vpc(this, 'RedshiftVpc');
const cluster = new redshift.Cluster(this, 'RedshiftCluster', {
vpc: vpc,
masterUser: {
masterUsername: 'admin'
},
removalPolicy: cdk.RemovalPolicy.DESTROY,
defaultDatabaseName: 'redshiftdb'
});
cluster.connections.allowDefaultPortFromAnyIpv4('Open to the world');
new cdk.CfnOutput(this, 'RedShiftClusterEndpoint', {
exportName: 'RedShiftClusterEndpoint',
value: cluster.clusterEndpoint.socketAddress
});
const user = new redshift.User(this, 'RedshiftUser', {
username: 'testredshiftuser',
cluster: cluster,
databaseName: 'redshiftdb',
});
const tableColumns: redshift.Column[] = [];
tableColumns.push(
{
name: 'column_0',
dataType: 'integer',
distKey: true,
comment: 'My #0 column'
}
);
for (var i = 1; i<= 1000; i++) {
tableColumns.push({
name: `column_${i}`,
dataType: 'integer',
distKey: false,
comment: `My #${i} column`
});
}
const table = new redshift.Table(this, 'RedshiftTable', {
cluster: cluster,
databaseName: 'redshiftdb',
tableColumns: tableColumns,
distStyle: redshift.TableDistStyle.KEY,
removalPolicy: cdk.RemovalPolicy.DESTROY,
sortStyle: redshift.TableSortStyle.AUTO,
tableName: 'my_redshift_table',
tableComment: 'My first table'
});
table.grant(user, redshift.TableAction.ALL);
}
}Gives the below error during deployment:
11:10:41 AM | CREATE_FAILED | Custom::RedshiftDatabaseQuery | RedshiftTable3DF2B3B1
Received response status [FAILED] from custom resource. Message returned: Error: RequestId: 1d0b3fd7-f29f-4b8b-905b-946d33155884 Error: Task timed out after 60.00 seconds
Logs: /aws/lambda/RedshiftCdkStack-QueryRedshiftDatabase3de5bea727da-7EJlN1dOMyCq
at invokeUserFunction (/var/task/framework.js:2:6)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async onEvent (/var/task/framework.js:1:369)
at async Runtime.handler (/var/task/cfn-response.js:1:1826) (RequestId: fbd6ca02-8a10-4f80-ae16-706f0241125a)
❌ RedshiftCdkStack failed: Error: The stack named RedshiftCdkStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: Error: RequestId: 1d0b3fd7-f29f-4b8b-905b-946d33155884 Error: Task timed out after 60.00 secondsLooking at the synthesized template, the below custom resource has timeout set to 60 (whereas other custom resources have timout of 900):
...
"QueryRedshiftDatabase3de5bea727da479686625efb56431b5f3DF81997": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": "cdk-hnb659fds-assets-139480602983-us-east-2",
"S3Key": "6bdd909f81c84ffe7d00cf4d6a2dbac8606429bcc05b0db3da842c1941a532f2.zip"
},
"Handler": "index.handler",
"Role": {
"Fn::GetAtt": [
"QueryRedshiftDatabase3de5bea727da479686625efb56431b5fServiceRole0A90D717",
"Arn"
]
},
"Runtime": "nodejs20.x",
"Timeout": 60
},
"DependsOn": [
"QueryRedshiftDatabase3de5bea727da479686625efb56431b5fServiceRoleDefaultPolicyDDD1388D",
"QueryRedshiftDatabase3de5bea727da479686625efb56431b5fServiceRole0A90D717"
],
"Metadata": {
"aws:cdk:path": "RedshiftCdkStack/Query Redshift Database3de5bea727da479686625efb56431b5f/Resource",
"aws:asset:path": "asset.6bdd909f81c84ffe7d00cf4d6a2dbac8606429bcc05b0db3da842c1941a532f2",
"aws:asset:is-bundled": false,
"aws:asset:property": "Code"
}
},
...Looks like the SingletonFunction used here in DatabaseQuery1 minute.
Perhaps the fix could be to expose handlerTimout property in DatabaseQueryHandlerProps which corresponds to Lambda function timeout property and use it's value to customize timeout. Unsure if this should be exposed at table level since end user should not be aware/concerned of any custom resource handlers used behind the scenes.
Describe the bug
The problem arises while trying to create a redshift table with many columns (in the order of the hundreds) using the Table construct.
During the deployment of the resource (by the means of a CodeBuild process issuing
cdk deploy -c env=test), I obtain the following error during the cloudformation CREATE action:The timeout error is related to a lambda generated automatically by the framework. Such timeout is by default set to 1 minute and I did not find a way to change it programmatically at runtime. Here's the code of the lambda in framework.js which raises the Exception:
Moreover, the table creation in redshfit is completed correctly (it looks like the CREATE TABLE instruction is issued and not aborted when the lambda fails), leaving the table untied to the cloudformation stack.
Here's an example of the code I am using:
Regression Issue
Last Known Working CDK Version
No response
Expected Behavior
I expect the table to be created successfully in redshift and the table to be part of the cloudformation stack.
Current Behavior
The table is created successfully in redshift but the table is not part of the cloudformation stack.
Reproduction Steps
Possible Solution
I think it is sufficient to increase the timeout for the lambda function
Additional Information/Context
No response
CDK CLI Version
2.150.0
Framework Version
No response
Node.js Version
18
OS
Ubuntu
Language
Python
Language Version
No response
Other information
I am using an EC2 instance with the following runtime provided in codebuild
aws/codebuild/standard:7.0 (Ubuntu standard:7.0)