benjicarpy19
commented
1 year ago Note: upgrading CDK to 2.64.0 does not resolve the issue
Open benjicarpy19 opened 1 year ago
benjicarpy19
commented
1 year ago Note: upgrading CDK to 2.64.0 does not resolve the issue
khushail
commented
1 year ago Hi @benjicarpy19 , thanks for reaching out. You might want to try the suggested solution provided in here.
Similar to what has been asked in the linked issue, could you share the repro code and if that's not feasible, it would be great to answer about stacks and resources.
benjicarpy19
commented
1 year ago Hi,
As mentioned I have already tried the solution in the link you suggested and I am still seeing the issue persist. Here is the full construct that is likely causing te memory issue. As you can see there are many lambdas all with node_modules which I believe is what is likely cuasing the memory outage. Just to re-iterate setting NODE_OPTIONS=--max_old_space_size=6000 before running cdk deploy does not resolve the issue
import { Duration, RemovalPolicy } from 'aws-cdk-lib';
import { ContentHandling, LambdaIntegration } from 'aws-cdk-lib/aws-apigateway';
import { IRole } from 'aws-cdk-lib/aws-iam';
import { Vpc } from 'aws-cdk-lib/aws-ec2';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as lambdaEventSources from 'aws-cdk-lib/aws-lambda-event-sources';
import { Runtime } from 'aws-cdk-lib/aws-lambda';
import { RestApiBase } from 'aws-cdk-lib/aws-apigateway';
import {
NodejsFunction,
NodejsFunctionProps
} from 'aws-cdk-lib/aws-lambda-nodejs';
import { Construct } from 'constructs';
import domains from '../config/domains';
import { ISecurityGroup } from 'aws-cdk-lib/aws-ec2';
import {
aws_events as events,
aws_events_targets as targets
} from 'aws-cdk-lib';
import { setRemovalPolicy } from '../utils/cdk-funcs';
interface LambdasApiProps {
environment: string;
userPoolId: string;
userPoolClientId: string;
defaultSecurityGroup: ISecurityGroup;
lambdaRole: IRole;
api: RestApiBase;
vpc: Vpc;
}
export default class LambdasApi extends Construct {
constructor(scope: Construct, id: string, props: LambdasApiProps) {
super(scope, id);
const { api } = props;
const runTime = Runtime.NODEJS_16_X;
const eventRule = new events.Rule(this, 'LambdaSchedule', {
schedule: events.Schedule.rate(Duration.minutes(1))
});
const priceListBucket = new s3.Bucket(this, 'OurBucket', {
autoDeleteObjects: true,
removalPolicy: RemovalPolicy.DESTROY
});
const orderPdfBucket = new s3.Bucket(this, 'OrderPDFBucket', {
removalPolicy: setRemovalPolicy(props.environment)
});
const s3PutEventSource = new lambdaEventSources.S3EventSource(
priceListBucket,
{
events: [s3.EventType.OBJECT_CREATED_PUT]
}
);
const nodeJsFunctionProps: NodejsFunctionProps = {
environment: {
AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
USER_POOL_ID: props.userPoolId,
USER_POOL_CLIENT_ID: props.userPoolClientId,
FRONTEND_URL: process.env.FRONTEND_URL as string,
DATABASE_URL: process.env.DATABASE_URL as string,
HUBSPOT_API_KEY: process.env.HUBSPOT_API_KEY as string,
API_BASE_URL: domains[props.environment].apiDomain as string,
STAGE: props.environment,
UPLOAD_PRICE_LIST_BUCKET: priceListBucket.bucketName,
UPLOAD_ORDER_PDF_BUCKET: orderPdfBucket.bucketName,
SENTRY_DSN: process.env.SENTRY_DSN as string
},
vpc: props.vpc,
role: props.lambdaRole,
runtime: runTime,
securityGroups: [props.defaultSecurityGroup],
memorySize: 1024,
timeout: Duration.seconds(900),
bundling: {
nodeModules: [
'@middy/http-json-body-parser',
'@middy/http-error-handler',
'@middy/core',
'@middy/http-cors',
'@middy/http-header-normalizer',
'@middy/util',
'@aws-sdk/client-cognito-identity-provider',
'@aws-sdk/client-secrets-manager',
'@prisma/client',
'@sentry/node',
'@hubspot/api-client',
'aws-jwt-verify',
'aws-sdk',
'prisma',
'read-excel-file',
'pdf-creator-node',
'cookie',
'dayjs',
'nodemailer',
'handlebars'
],
commandHooks: {
beforeBundling(_inputDir: string, _outputDir: string) {
return [];
},
beforeInstall(inputDir: string, outputDir: string) {
const commands = [
`cp -R ${inputDir}/prisma ${outputDir}/`,
`mkdir -p ${outputDir}/src/lambda/templates`,
`cp -R ${inputDir}/src/lambda/templates/pdf ${outputDir}/src/lambda/templates`
];
if (process.env.NODE_ENV !== 'CI') {
commands.push(`cp -R ${inputDir}/.env ${outputDir}/`);
}
return commands;
},
afterBundling(_inputDir: string, outputDir: string) {
return [
`cd ${outputDir}`,
`prisma generate`,
`rm -rf node_modules/@prisma/engines`,
`rm -rf node_modules/@prisma/client/node_modules node_modules/.bin node_modules/prisma`
];
}
}
}
};
// status
const statusLambda = new LambdaIntegration(
new NodejsFunction(this, 'statusFunction', {
entry: 'dist/src/lambda/resolvers/status.js',
functionName: 'status',
...nodeJsFunctionProps
})
);
// admin
const adminGetUserLambda = new LambdaIntegration(
new NodejsFunction(this, 'adminGetUserFunction', {
entry: 'dist/src/lambda/resolvers/admin/user/get-user.js',
functionName: 'admin_user_get-user',
...nodeJsFunctionProps
})
);
const adminGetProductsUploadLambda = new LambdaIntegration(
new NodejsFunction(this, 'adminGetProductsUploadLambdaFunction', {
entry:
'dist/src/lambda/resolvers/admin/products-upload/get-products-upload.js',
functionName: 'admin_products-upload_get-products-upload',
...nodeJsFunctionProps
})
);
const adminPostProductsUploadExecuteLambda = new NodejsFunction(
this,
'adminPostProductsUploadExecuteLambdaFunction',
{
entry:
'dist/src/lambda/resolvers/admin/products-upload/post-products-upload-execute.js',
functionName: 'admin_products-upload_post-products-upload-execute',
...nodeJsFunctionProps
}
);
const adminPostProductsUploadLambda = new LambdaIntegration(
new NodejsFunction(this, 'adminPostProductsUploadLambdaFunction', {
entry:
'dist/src/lambda/resolvers/admin/products-upload/post-products-upload.js',
functionName: 'admin_products-upload_post-products-upload',
...nodeJsFunctionProps
})
);
adminPostProductsUploadExecuteLambda.addEventSource(s3PutEventSource);
// Auth
const loginLambda = new LambdaIntegration(
new NodejsFunction(this, 'loginFunction', {
entry: 'dist/src/lambda/resolvers/auth/login.js',
functionName: 'auth_login',
...nodeJsFunctionProps
})
);
const registerConfirmLambda = new LambdaIntegration(
new NodejsFunction(this, 'registerConfirmFunction', {
entry: 'dist/src/lambda/resolvers/auth/register-confirm.js',
functionName: 'auth_register-confirm',
...nodeJsFunctionProps
})
);
const forgotPasswordLambda = new LambdaIntegration(
new NodejsFunction(this, 'forgotPasswordLambda', {
entry: 'dist/src/lambda/resolvers/auth/forgot-password.js',
functionName: 'auth_forgot-password',
...nodeJsFunctionProps
})
);
const confirmForgotPasswordLambda = new LambdaIntegration(
new NodejsFunction(this, 'confirmForgotPasswordLambda', {
entry: 'dist/src/lambda/resolvers/auth/confirm-forgot-password.js',
functionName: 'auth_confirm-forgot-password',
...nodeJsFunctionProps
})
);
const resendConfirmationCodeLambda = new LambdaIntegration(
new NodejsFunction(this, 'resendConfirmationCodeLambda', {
entry: 'dist/src/lambda/resolvers/auth/resend-confirmation-code.js',
functionName: 'auth_resend-confirmation-code',
...nodeJsFunctionProps
})
);
const logoutLambda = new LambdaIntegration(
new NodejsFunction(this, 'logoutLambda', {
entry: 'dist/src/lambda/resolvers/auth/logout.js',
functionName: 'auth_logout',
...nodeJsFunctionProps
})
);
// order
const createOrderLambda = new LambdaIntegration(
new NodejsFunction(this, 'createOrderFunction', {
entry: 'dist/src/lambda/resolvers/order/create-order.js',
functionName: 'order_create-order',
...nodeJsFunctionProps
})
);
const updateOrderLambda = new LambdaIntegration(
new NodejsFunction(this, 'updateOrderFunction', {
entry: 'dist/src/lambda/resolvers/order/update-order.js',
functionName: 'order_update-order',
...nodeJsFunctionProps
})
);
const deleteOrderLambda = new LambdaIntegration(
new NodejsFunction(this, 'deleteOrderFunction', {
entry: 'dist/src/lambda/resolvers/order/delete-order.js',
functionName: 'order_delete-order',
...nodeJsFunctionProps
})
);
const getOrderLambda = new LambdaIntegration(
new NodejsFunction(this, 'getOrderFunction', {
entry: 'dist/src/lambda/resolvers/order/get-order.js',
functionName: 'order_get-order',
...nodeJsFunctionProps
})
);
// wholesaler order
const createWholesalerOrderLambda = new LambdaIntegration(
new NodejsFunction(this, 'createWholesalerOrderFunction', {
entry:
'dist/src/lambda/resolvers/wholesaler-order/create-wholesaler-order.js',
functionName: 'wholesaler-order_create-wholesaler-order',
...nodeJsFunctionProps
})
);
const updateWholesalerOrderLambda = new LambdaIntegration(
new NodejsFunction(this, 'updateWholesalerOrderFunction', {
entry:
'dist/src/lambda/resolvers/wholesaler-order/update-wholesaler-order.js',
functionName: 'wholesaler-order_update-wholesaler-order',
...nodeJsFunctionProps
})
);
// PDF order
const getPdfWholesalerOrderLambda = new LambdaIntegration(
new NodejsFunction(this, 'getPdfWholesalerOrderFunction', {
entry:
'dist/src/lambda/resolvers/wholesaler-order/pdf/get-wholesaler-order-pdf.js',
functionName: 'wholesaler-order_pdf_get-wholesaler-order-pdf',
...nodeJsFunctionProps
})
);
// orders
const getOrdersLambda = new LambdaIntegration(
new NodejsFunction(this, 'getOrdersFunction', {
entry: 'dist/src/lambda/resolvers/orders/get-orders.js',
functionName: 'orders_get-orders',
...nodeJsFunctionProps
})
);
// product-filters
const getProductFiltersLambda = new LambdaIntegration(
new NodejsFunction(this, 'getProductFiltersFunction', {
entry:
'dist/src/lambda/resolvers/product-filters/get-product-filters.js',
functionName: 'product-filters_get-product-filters',
...nodeJsFunctionProps
})
);
// user
const createMeLambda = new LambdaIntegration(
new NodejsFunction(this, 'createMeFunction', {
entry: 'dist/src/lambda/resolvers/user/me/create-me.js',
functionName: 'user_me_create-me',
...nodeJsFunctionProps
})
);
const getMeLambda = new LambdaIntegration(
new NodejsFunction(this, 'getMeLambda', {
entry: 'dist/src/lambda/resolvers/user/me/get-me.js',
functionName: 'user_me_get-me',
...nodeJsFunctionProps
})
);
const updateMeLambda = new LambdaIntegration(
new NodejsFunction(this, 'updateMeLambda', {
entry: 'dist/src/lambda/resolvers/user/me/update-me.js',
functionName: 'user_me_update-me',
...nodeJsFunctionProps
})
);
//venue
const createVenueLambda = new LambdaIntegration(
new NodejsFunction(this, 'createVenueLambda', {
entry: 'dist/src/lambda/resolvers/venue/create-venue.js',
functionName: 'venue_create-venue',
...nodeJsFunctionProps
})
);
const updateVenueLambda = new LambdaIntegration(
new NodejsFunction(this, 'updateVenueLambda', {
entry: 'dist/src/lambda/resolvers/venue/update-venue.js',
functionName: 'venue_update-venue',
...nodeJsFunctionProps
})
);
const getVenueLambda = new LambdaIntegration(
new NodejsFunction(this, 'getVenueLambda', {
entry: 'dist/src/lambda/resolvers/venue/get-venue.js',
functionName: 'venue_get-venue',
...nodeJsFunctionProps
})
);
const deleteVenueLambda = new LambdaIntegration(
new NodejsFunction(this, 'deleteVenueLambda', {
entry: 'dist/src/lambda/resolvers/venue/delete-venue.js',
functionName: 'venue_delete-venue',
...nodeJsFunctionProps
})
);
const getVenuesLambda = new LambdaIntegration(
new NodejsFunction(this, 'getVenuesLambda', {
entry: 'dist/src/lambda/resolvers/venues/get-venues.js',
functionName: 'venues_get-venues',
...nodeJsFunctionProps
})
);
const getVenuesCountLambda = new LambdaIntegration(
new NodejsFunction(this, 'getVenuesCount', {
entry: 'dist/src/lambda/resolvers/venues/count/get-venues-count.js',
functionName: 'venues_count_get-venues-count',
...nodeJsFunctionProps
})
);
// wholesaler
const createLinkedWholesalerLambda = new LambdaIntegration(
new NodejsFunction(this, 'createLinkedWholesalerLambda', {
entry:
'dist/src/lambda/resolvers/wholesaler/linked/create-linked-wholesaler.js',
functionName: 'wholesaler_linked_create-linked-wholesaler',
...nodeJsFunctionProps
})
);
const updateLinkedWholesalerLambda = new LambdaIntegration(
new NodejsFunction(this, 'updateLinkedWholesalerLambda', {
entry:
'dist/src/lambda/resolvers/wholesaler/linked/update-linked-wholesaler.js',
functionName: 'wholesaler_linked_update-linked-wholesaler',
...nodeJsFunctionProps
})
);
const deleteLinkedWholesalerLambda = new LambdaIntegration(
new NodejsFunction(this, 'deleteLinkedWholesalerLambda', {
entry:
'dist/src/lambda/resolvers/wholesaler/linked/delete-linked-wholesaler.js',
functionName: 'wholesaler_linked_delete-linked-wholesaler',
...nodeJsFunctionProps
})
);
const getLinkedWholesalersLambda = new LambdaIntegration(
new NodejsFunction(this, 'getLinkedWholesalersLambda', {
entry:
'dist/src/lambda/resolvers/wholesalers/linked/get-linked-wholesalers.js',
functionName: 'wholesalers_linked_get-linked-wholesalers',
...nodeJsFunctionProps
})
);
const getWholesalersLambda = new LambdaIntegration(
new NodejsFunction(this, 'getWholesalersLambda', {
entry: 'dist/src/lambda/resolvers/wholesalers/get-wholesalers.js',
functionName: 'wholesalers_get-wholesalers',
...nodeJsFunctionProps
})
);
// product
const getProductsFunction = new NodejsFunction(this, 'getProductsLambda', {
entry: 'dist/src/lambda/resolvers/products/get-products.js',
functionName: 'products_get-products',
...nodeJsFunctionProps
});
const getProductsLambda = new LambdaIntegration(getProductsFunction);
// below keeps the lambda warm
eventRule.addTarget(new targets.LambdaFunction(getProductsFunction));
// product savings
const getProductsSavingsLambda = new LambdaIntegration(
new NodejsFunction(this, 'getProductsSavingsLambda', {
entry:
'dist/src/lambda/resolvers/products/savings/get-products-savings.js',
functionName: 'products_savings_get-products-savings',
...nodeJsFunctionProps
})
);
const createProductSavingsLambda = new LambdaIntegration(
new NodejsFunction(this, 'createProductSavingsLambda', {
entry:
'dist/src/lambda/resolvers/product/savings/create-product-savings.js',
functionName: 'product_savings_create-product-savings',
...nodeJsFunctionProps
})
);
const deleteProductSavingsLambda = new LambdaIntegration(
new NodejsFunction(this, 'deleteProductSavingsLambda', {
entry:
'dist/src/lambda/resolvers/product/savings/delete-product-savings.js',
functionName: 'product_savings_delete-product-savings',
...nodeJsFunctionProps
})
);
const updateProductSavingsLambda = new LambdaIntegration(
new NodejsFunction(this, 'updateProductSavingsLambda', {
entry:
'dist/src/lambda/resolvers/product/savings/update-product-savings.js',
functionName: 'product_savings_update-product-savings',
...nodeJsFunctionProps
})
);
const authResource = api.root.addResource('auth');
const adminResource = api.root.addResource('admin');
const adminUserResource = adminResource.addResource('user');
const adminProductsUploadResource =
adminResource.addResource('products-upload');
const adminUserResourceWithId = adminUserResource.addResource('{id}');
const loginResource = authResource.addResource('login');
const registerConfirmResource =
authResource.addResource('register-confirm');
const forgotPasswordResource = authResource.addResource('forgot-password');
const confirmForgotPasswordResource = authResource.addResource(
'confirm-forgot-password'
);
const resendConfirmationCodeResource = authResource.addResource(
'resend-confirmation-code'
);
const logoutResource = authResource.addResource('logout');
const productFiltersResource = api.root.addResource('product-filters');
const userResource = api.root.addResource('user');
const meResource = userResource.addResource('me');
const venueResource = api.root.addResource('venue');
const venueResourceWithId = venueResource.addResource('{id}');
const venuesResource = api.root.addResource('venues');
const venuesResourceCount = venuesResource.addResource('count');
const orderResource = api.root.addResource('order');
const orderResourceWithId = orderResource.addResource('{id}');
const wholesalerOrderResource =
orderResourceWithId.addResource('wholesaler-order');
const wholesalerOrderResourceWithId = wholesalerOrderResource.addResource(
'{wholesalerOrderId}'
);
const wholesalerOrderPdfResource =
wholesalerOrderResource.addResource('pdf');
const ordersResource = api.root.addResource('orders');
const wholesalerResource = api.root.addResource('wholesaler');
const linkedWholesalerResource = wholesalerResource.addResource('linked');
const linkedWholesalerResourceWithId =
linkedWholesalerResource.addResource('{id}');
const wholesalersResource = api.root.addResource('wholesalers');
const linkedWholesalersResource = wholesalersResource.addResource('linked');
const productsResource = api.root.addResource('products');
const productsSavingsResource = productsResource.addResource('savings');
const productResource = api.root.addResource('product');
const productSavingsResource = productResource.addResource('savings');
const productSavingsResourceWithId =
productSavingsResource.addResource('{id}');
api.root.addMethod('GET', statusLambda);
adminUserResourceWithId.addMethod('GET', adminGetUserLambda);
adminProductsUploadResource.addMethod('GET', adminGetProductsUploadLambda);
adminProductsUploadResource.addMethod(
'POST',
adminPostProductsUploadLambda
);
if (process.env.STAGE === 'local') {
const localAdminProductsUploadResource =
adminProductsUploadResource.addResource('local');
localAdminProductsUploadResource.addMethod(
'POST',
new LambdaIntegration(adminPostProductsUploadExecuteLambda)
);
}
loginResource.addMethod('POST', loginLambda);
registerConfirmResource.addMethod('POST', registerConfirmLambda);
forgotPasswordResource.addMethod('POST', forgotPasswordLambda);
confirmForgotPasswordResource.addMethod(
'POST',
confirmForgotPasswordLambda
);
resendConfirmationCodeResource.addMethod(
'POST',
resendConfirmationCodeLambda
);
logoutResource.addMethod('POST', logoutLambda);
productFiltersResource.addMethod('GET', getProductFiltersLambda);
meResource.addMethod('POST', createMeLambda);
meResource.addMethod('GET', getMeLambda);
meResource.addMethod('PUT', updateMeLambda);
venueResource.addMethod('POST', createVenueLambda);
venueResourceWithId.addMethod('GET', getVenueLambda);
venueResourceWithId.addMethod('PUT', updateVenueLambda);
venueResourceWithId.addMethod('DELETE', deleteVenueLambda);
venuesResource.addMethod('GET', getVenuesLambda);
venuesResourceCount.addMethod('GET', getVenuesCountLambda);
orderResource.addMethod('POST', createOrderLambda);
orderResourceWithId.addMethod('DELETE', deleteOrderLambda);
orderResourceWithId.addMethod('PUT', updateOrderLambda);
orderResourceWithId.addMethod('GET', getOrderLambda);
wholesalerOrderResource.addMethod('POST', createWholesalerOrderLambda);
wholesalerOrderResourceWithId.addMethod('PUT', updateWholesalerOrderLambda);
wholesalerOrderPdfResource.addMethod('GET', getPdfWholesalerOrderLambda);
ordersResource.addMethod('GET', getOrdersLambda);
linkedWholesalerResource.addMethod('POST', createLinkedWholesalerLambda);
linkedWholesalerResourceWithId.addMethod(
'PUT',
updateLinkedWholesalerLambda
);
linkedWholesalerResourceWithId.addMethod(
'DELETE',
deleteLinkedWholesalerLambda
);
linkedWholesalersResource.addMethod('GET', getLinkedWholesalersLambda);
wholesalersResource.addMethod('GET', getWholesalersLambda);
productsResource.addMethod('GET', getProductsLambda);
productsSavingsResource.addMethod('GET', getProductsSavingsLambda);
productSavingsResource.addMethod('POST', createProductSavingsLambda);
productSavingsResourceWithId.addMethod(
'DELETE',
deleteProductSavingsLambda
);
productSavingsResourceWithId.addMethod('PUT', updateProductSavingsLambda);
}
}
RomainMuller
commented
1 year ago Is NODE_OPTIONS=--max_old_space_size=6000 && cdk deploy the actual command you ran? If so, you actually haven't changed the max old space size (your screenshot suggests the Node VM hit heap limit at 2G, which is the default, and would be consistent with this observation), due to the presence of the && there... I'm actually surprised this didn't err out (but this is beyond the point)...
You could try:
$ NODE_OPTIONS=--max_old_space_size=6000 cdk deployor
$ export NODE_OPTIONS=--max_old_space_size=6000
$ cdk deployOh interesting, you could well be right. Yes that is the actual command I was running. Is there a way to log overall memory usage after the cdk deploy process?
RomainMuller
commented
1 year ago No I don't think there is such a provision in place today.
Your app could actually just periodically dump JS heap usage information out to STDERR or to a file, using a combination of setInterval and node:v8.getHeapSpaceStatistics(). You might want to call .unref() on the object returned by setInterval so that it does not prevent the Node VM from exiting once other work is done.
benjicarpy19
commented
1 year ago Thanks, that could be worth running alongside the deploy operation. I am curious though as to why it is/ was timing out. If the default max_old_space_size is 2G as you mentioned and my runner has 8G of memory, how was it running out of memory? Surely it should have never surpassed 2G if the max_old_space_size was not set properly and therefore not run out of memory as it would have never surpsassed 8G?
Describe the bug
I am running into a "JavaScript heap out of memory" error when running
cdk deployin my CI. I am using at3.largeinstance which has 8GiB memory as shown here.I searched for this issue and it was mentioned that setting the
NODE_OPTIONS=--max_old_space_sizeprior to runningcdk deployshould fix the memory issue however when I amend thecdk deploycommand to be like so:NODE_OPTIONS=--max_old_space_size=6000 && cdk deployI still get the JavaScript heap out of memory issue even though I am limiting it to 6000 when my available memory should be 8000
Expected Behavior
I would expect with my defined node memory limit it should not run out of memory
Current Behavior
It currently runs out of memory while it appears to be creating the cloud formation template. Please see image below for reference in my CI
Reproduction Steps
I am unable to easily provide this i'm afraid, I am also not sure it is required to identify the issue. What I can provide is my general lambda configuration that causes the memory issue. I believe it is likely due to
node_modulesacross 40 or so different lambdas creating large memory consumption. Unfortunately due to my setup it would be very difficult to migrate to using a lambda llayer to abstract the node_modules away from the lambda function asset bundles.Possible Solution
I did try using the
-no-asset-parallelismflag however this makes deployment painfully slow as no assest run in parallel at all.Additional Information/Context
No response
CDK CLI Version
2.59.0
Framework Version
No response
Node.js Version
16.3.0
OS
Linux
Language
Typescript
Language Version
No response
Other information
No response