search

aws / jsii

jsii allows code in any language to naturally interact with JavaScript classes. It is the technology that enables the AWS Cloud Development Kit to deliver polyglot libraries from a single codebase!
https://aws.github.io/jsii
Apache License 2.0
2.62k stars 244 forks source link

Increase heap memory for jsii engine #745

Closed bverhoeve closed 1 year ago

bverhoeve commented 5 years ago

:rocket: Feature Request

When using JSII for small AWS CDK applications, the JSII kernel will normally not run out of heap memory. However, for larger apps (>800 resources, multiple stacks), depending on the complexity, you can encounter the following error: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory.

Is there already a way to increase the Javascript heap memory beyond the default value?

Affected Languages

General Information

Description

Add a setting in the cdk.json file which specifies the maximum allowed heap memory that jsii can allocate. Set the maximum heap memory then to this setting.

Proposed Solution

RomainMuller commented 5 years ago

I don't think we can host this setting in cdk.json (because JSII can be used in contexts other than cdk).

Ability to customize the node runtime is however a very interesting idea and we should dig into this further. We should also consider what other aspects of the node runtime we could/should allow users to control.

RomainMuller commented 5 years ago

Note that in the meantime, you can pass additional arguments to the node process by setting the NODE_OPTIONS environment variable. In particular, controlling the max heap size can be achieved using:

# Sets the maximum heap to 4GiB
NODE_OPTIONS='--max-old-space-size=4096'
bverhoeve commented 5 years ago

Does jsii always use the system node installation to run? Or does it use another kernel?

If the system node is the case, the NODE_OPTIONS is quite nice already as a workaround :)

bverhoeve commented 4 years ago

@RomainMuller the NODE_OPTIONS variable works for increasing the max heap size, but unfortunately not the call stack size. This can be changed by using node with the --stack_size option set to a certain value. Unfortunately, the --stack_size option can't be set yet using NODE_OPTIONS.

RomainMuller commented 4 years ago

Ah - an alternate way to solve this until we actually support acting on those settings via configuration is to create a wrapper around node that is ahead of the real node in your $PATH, something along the lines of (warning: I haven't tested it):

#!/bin/bash

/usr/local/bin/node --stack_size=4096 $@
ansgarm commented 3 years ago

We recently encountered this as well in the CDK for Terraform when generating provider bindings for the AWS Terraform Provider for Go.

Here is a small example which can be used to reproduce the issue or to do some profiling. For now we went the route of setting NODE_OPTIONS on process.env so it is made available to the child process for jsii-pakmak which is spawned by jsii-srcmak (which we use). While this seems like an okayish fix, I'm still a bit confused why the memory usage is that heavy.

Do you have any pointers where one might start digging, @RomainMuller? We're experiencing this particularly with Go (to this 7 GB extent 🏋️😄)

github-actions[bot] commented 1 year ago

This issue has not received any attention in 2 years. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.