Javascript heap out of memory

[roderik]

After a while of fairly heavy use (like we use a Parity node in production) Ganache stops working due to an out of memory error. (docker image: trufflesuite/ganache-cli:v6.1.8)

ethereum_1                | 
ethereum_1                | <--- Last few GCs --->
ethereum_1                | 
ethereum_1                | [1:0x561ab041e000] 26545867 ms: Mark-sweep 1404.1 (1464.4) -> 1404.0 (1464.4) MB, 378.3 / 0.0 ms  allocation failure GC in old space requested
ethereum_1                | [1:0x561ab041e000] 26546370 ms: Mark-sweep 1404.0 (1464.4) -> 1404.0 (1433.4) MB, 502.1 / 0.1 ms  last resort GC in old space requested
ethereum_1                | [1:0x561ab041e000] 26546865 ms: Mark-sweep 1404.0 (1433.4) -> 1404.0 (1433.4) MB, 495.8 / 0.1 ms  last resort GC in old space requested
ethereum_1                | 
ethereum_1                | 
ethereum_1                | <--- JS stacktrace --->
ethereum_1                | 
ethereum_1                | ==== JS stack trace =========================================
ethereum_1                | 
ethereum_1                | Security context: 0x56ffe825879 <JSObject>
ethereum_1                |     1: set(this=0x912f019afe1 <Map map = 0x3997f91848d9>,0x13318c210739 <JSArray[152]>,0xc4b8d949dd9 <JSArray[152]>)
ethereum_1                |     2: set [/src/build/cli.node.js:~10] [pc=0x1d0b12cec68b](this=0x912f019aeb1 <yc map = 0x236896533a01>,e=0x13318c210739 <JSArray[152]>,a=0xc4b8d949dd9 <JSArray[152]>)
ethereum_1                |     3: Ic [/src/build/cli.node.js:~10] [pc=0x1d0b12faf924](this=0x382ee240c209 <JSGlobal Object>,e=0x13318c210739...
ethereum_1                | 
ethereum_1                | FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

To give an idea what I mean by fairly heavy usage, the full log: https://gist.github.com/roderik/82cba1882eb67b7e64c6482d12dd25d5 (about 19k lines)

Startup command:

  ethereum:
    image: trufflesuite/ganache-cli:v6.1.8
    ports:
      - '8545:8545'
    command:
      [
        'node',
        './build/cli.node.js',
        '--secure',
        '--gasPrice',
        '0',
        '--unlock',
        '0',
        '--gasLimit',
        '0xfffffffffff',
        '--host',
        '0.0.0.0',
        '--networkId',
        '1337',
        '--mnemonic',
        'robot robot robot robot robot robot robot robot robot robot robot robot',
      ]

[benjamincburns]

Thanks for reporting this one @roderik! We'll be sure to look into it quickly!

[JohannesGitHub]

I got a similar error message right after executing truffle debug. (Ganache CLI v6.1.8 (ganache-core: 2.2.1))

<--- Last few GCs --->

[46951:0x104001200]   208749 ms: Scavenge 1396.6 (1425.0) -> 1396.2 (1426.5) MB, 3.0 / 0.0 ms  (average mu = 0.177, current mu = 0.068) allocation failure 
[46951:0x104001200]   209098 ms: Mark-sweep 1397.2 (1426.5) -> 1396.5 (1426.5) MB, 344.8 / 0.0 ms  (average mu = 0.120, current mu = 0.058) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x2948aa04fb7d]
Security context: 0x345feb79d969 <JSObject>
    1: s [0x345f5c723f51] [/Users/user/.nvm/versions/node/v11.0.0/lib/node_modules/ganache-cli/build/cli.node.js:~30] [pc=0x2948aa7b29a0](this=0x345f5ce2f1e9 <v map = 0x345ffa1cd1b1>,0x345ff6aef721 <Object map = 0x345f331e7f61>,0x345ff6aefb89 <JSFunction (sfi = 0x345f329e9421)>)
    2: /* anonymous */(aka /* anonymous */) [0x345ff6aef849] [/Users/user...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

 1: 0x10003a9d9 node::Abort() [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
 2: 0x10003abe4 node::FatalTryCatch::~FatalTryCatch() [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
 3: 0x10019ed17 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
 4: 0x10019ecb4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
 5: 0x1005a5882 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
 6: 0x1005a4838 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
 7: 0x1005a2443 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
 8: 0x1005aecbc v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
 9: 0x1005aed3f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
10: 0x10057dfc4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
11: 0x100832070 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/user/.nvm/versions/node/v11.0.0/bin/node]
12: 0x2948aa04fb7d 
Abort trap: 6

[benjamincburns]

@JohannesGitHub can you please raise a separate issue for that? FWIW, it's possible for debug_traceTransaction to consume a lot of memory when debugging high gas-cost transactions. We might someday implement a streaming version of debug_traceTransaction, but in the mean time you'll need to increase node's heap size limits when using ganache for debugging purposes.

[benjamincburns]

@roderik is this still a problem for you with ganache-cli@6.2.1? Also if it is still a problem, by any chance would it be possible for you to publish a repro log with the --verbose flag enabled? This will let us write a parser that will replay your session locally so we can get a better view into it. So far with latest ganache, I'm not able to get it to exceed a 191M resident set size on my machine.

@eshaben has also had trouble reproducing it on her machine with ganache-cli@6.1.8, so I suspect we are missing something in our reproduction attempts.

[benjamincburns]

@roderik I also see a bunch of calls to trace_replayTransaction - I don't think we support this call. Are you running a modified version?

[benjamincburns]

Actually, @JohannesGitHub, I think trufflesuite/ganache#449 is the issue you'll want to track.

[roderik]

@benjamincburns switched away from ganache for these workloads, only use it for contract development atm.

[davidmurdoch]

Closing as we've been unable to reproduce on ganache-cli@6.2.3.

[brockelmore]

I am encountering this when using ganache-cli@6.7.0 in docker mode and using

gananche-cli --fork https://mainnet.infura.io/v3/{project_id} --networkId 1001 --gasPrice 0x1312D00 -e 100000000000000000000 --gasLimit 8000000 

for the creation of ganache. Then I would run truffle debug --network test 0x98ace240d67339e57e588e1c902102d1b0dde67bd335d16a6467b24d3bd841f9. It compiles the contracts, then fails to get information about the project and tx.

I've tried increasing --max-old-space-size=8192, but to no avail.

Here is the output:

debug_traceTransaction
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2019-10-26T20:18:00.144401000Z 
<--- Last few GCs --->
2019-10-26T20:18:00.144603700Z 
[1:0x558e19cdeaa0]   414610 ms: Mark-sweep 1397.8 (1427.2) -> 1397.0 (1427.2) MB, 371.0 / 0.0 ms  (average mu = 0.124, current mu = 0.039) allocation failure scavenge might not succeed
2019-10-26T20:18:00.144674800Z 
2019-10-26T20:18:00.144704000Z 
<--- JS stacktrace --->
2019-10-26T20:18:00.144884100Z 

[davidmurdoch]

Thanks @brockelmore for the reproduction steps. Our debug_traceTransaction is a bit expensive to run and this reproduction will be very helpful in our optimization!

[brockelmore]

Update on this: above tx was with a local smart contract deployed. Here is a tx that should fail that is on mainnet and with the fork should fail for you as well. 0x985b3c7ed63ecb30e9de94b8ab7377a42c86643536a512ec6331874d6020d127

I have no conception for how the underlying is processed, but the above tx interacts with a contract that uses ABIEncoderV2, so I am not sure if that has anything to do with it.

[thegostep]

I am getting this error when a solidity function gets stuck in an infinite while loop.

[brockelmore]

Shouldn't it just run until its out of gas? Why would an infinite loop cause the error?

[thegostep]

Depends on the ganache evm implementation - seems like there could be a bug in gas accounting

[brockelmore]

Turns out it happens very reproducibly when you perform a snapshot prior to debugging as well it seems. Also ganache-cli —max-old-space-size didn’t seem to actually change the heap size. export NODE_OPTIONS=“ —max-old-space-size=8192” does though. Also seems that there are timeout issues with the provider if you don’t set it manually to be extremely long. All that said, got it to a working state 30% of the time

[fubhy]

I can also verify that there is definitely a memory leak in ganache. I've also observed it while tracking the it with node --inspect in and profiling the memory allocation in chrome devtools (capturing >1 day of data). Memory usage steadily increases including some inexplicable spikes in between. I haven't yet found the culprit though.

[m1cm1c]

The problem also occurs during a gas estimation (of a large tx) with version 6.12.1:

[...]
eth_getBalance
eth_accounts
eth_estimateGas

<--- Last few GCs --->
ti[44175:0x4c28460]   330465 ms: Mark-sweep 2047.3 (2050.6) -> 2046.4 (2050.6) MB, 1388.5 / 0.0 ms  (+ 0.0 ms in 15 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1391 ms) (average mu = 0.063, current mu = 0.002) allocati[44175:0x4c28460]   331929 ms: Mark-sweep 2047.4 (2050.6) -> 2046.5 (2050.6) MB, 1462.3 / 0.0 ms  (+ 0.0 ms in 15 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1463 ms) (average mu = 0.032, current mu = 0.001) allocati

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa2afd0 node::Abort() [node]
 2: 0x97a467 node::FatalError(char const*, char const*) [node]
 3: 0xb9e04e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9e3c7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd3e7d5  [node]
 6: 0xd3f17b v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd4cf72 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd4ddc5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd5078c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd1fe2b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x105014f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x13a9e39  [node]
Aborted (core dumped)
[christoph@ladybug ~]$ ganache-cli --version
Ganache CLI v6.12.1 (ganache-core: 2.13.1)

At the time of the crash, my machine is nowhere near out of memory. Node consumes about 2.3 GB at the time of the crash (the dip in memory at the very right where the CPU load also drops suddenly).

This crash is reproducible. If you want to reproduce it, clone https://github.com/BlockInfinity/kontaktdatenblattOnBlockchain (recursively), open CertificateVerification.sol in Remix, start Ganache CLI with a high block gas limit (I used 80 million in most of my attempts but 30 should suffice), deploy CertificateVerification using this argument:

0x308203c030820366a00302010202084cdaaea738605ce2300a06082a8648ce3d040302304d311e301c06035504031315542d53797374656d732d456e6572677943412e434131123010060355040a1309534d2d504b492d4445310b3009060355040613024445310a30080603550405130131301e170d3139303830313131353230385a170d3231303830313233353935395a30819e31193017060355040313103530686572747a2e454d542e70726f6431123010060355040a1309534d2d504b492d444531223020060355040b13193530486572747a205472616e736d697373696f6e20476d6248310b3009060355040613024445310a30080603550405130132310e300c060355041113053130353537310f300d060355040713064265726c696e310f300d060355040813064265726c696e305a301406072a8648ce3d020106092b240303020801010703420004469eabb5505638c958de8f705489d8b7a6241db17889758cf2cb143a6bdfcb02808af47f11aad57e2cf3dd9fde84a5c8b9bc55a7b89a99e9021bcc898fc42a65a38201db308201d7301f0603551d23041830168014807b332a0ebad39fcdf3ec33d10157ad762ea957301d0603551d0e041604144d8c9619a52f37625f0f498abd5d7c3f99e29d0b300e0603551d0f0101ff040403020780303d0603551d2004363034303206092b06010401bd470d243025302306082b0601050507020116177777772e74656c657365632e64652f656e65726779636130210603551d11041a30188116706f73746d6173746572403530686572747a2e636f6d30490603551d1204423040811e456e6572677943415f537570706f727440742d73797374656d732e636f6d861e687474703a2f2f7777772e74656c657365632e64652f656e657267796361300c0603551d130101ff040230003081c90603551d1f0481c13081be3081bba081b8a081b5863f687474703a2f2f63726c2e656e6572677963612e74656c657365632e64652f726c2f542d53797374656d732d456e6572677943412e43415f534e312e63726c86726c6461703a2f2f6c6461702e656e6572677963612e74656c657365632e64652f73657269616c6e756d6265723d312c434e3d542d53797374656d732d456e6572677943412e43412c4f3d534d2d504b492d44452c433d44453f43657274696669636174655265766f636174696f6e4c697374300a06082a8648ce3d0403020348003045022100932ccefda0386c184deb40c256fb87695d35b2c860f56c6adc170821f08ff6be022053108d2516b974e7a438ff4f4e1483c406242f6ec07dc86e1546e847abc31532

And then simply call performVerification(). Ganache CLI will crash about one minute after Remix (automatically) started the gas estimation. You might have to increase the gas limit in Remix for this to happen.

[kevinweaver]

@m1cm1c Thank you for making an easily reproduce-able repo! The issue you're describing is a bit different than the initial one opened in this thread. If you are still experiencing this, do you mind opening a new issue?

As for the original issue, upgrading to ganache@7.0.0-alpha should fix this. Closing the issue, please open a new issue if upgrading does not solve this.

[m1cm1c]

@kevinweaver Both the transaction and the gas estimation work with Ganache v7.0.0-alpha.1. Thank you for the hint about upgrading. :)

[kifah1989]

i used bun to run ganache-cli commands. i used to have memory heap issues when calling solidity functions but when i used bun the error seems to go away. i made a script in linux to run ganache like this.

!/usr/bin/env bun

ganache

[davidmurdoch]

@kifah1989 that's cool. I didn't know it could be run via bun! Wonder is we can add native support for it in the package so you won't need to create that bun script 🤔 .

But im mostly interested in the fact that you are still having memory issues. Can you open a new issue describing the problem and how to reproduce it?

[kifah1989]

i'm using two ganache-cli each with two different chain id to simulate network change. terminal one bun run ganache-cli --port 8545 --chain.chainId 1337 --db eth_db --mnemonic "inner goddess spoil verify ostrich crazy good stage federal type wall wheel" --accounts 5 --defaultBalanceEther 10000 --fork https://eth-goerli.g.alchemy.com/v2/<api key>

terminal two bun run ganache-cli --port 8546 --chain.chainId 1338 --db arb_db --mnemonic "inner goddess spoil verify ostrich crazy good stage federal type wall wheel" --accounts 5 --defaultBalanceEther 10000 --fork https://arb-goerli.g.alchemy.com/v2/<api key>

most of the heavy work is done in the rpc of terminal two and used to crash with memory heap error.

i wanted to create the same scenario with ganach-ui but it doesn't support changing the chainid in a workspace.

[davidmurdoch]

What kind of work is being done? Sending transactions, eth_call, debug_traceTransaction, ...?

[kifah1989]

eth_getLogs

[davidmurdoch]

@kifah1989 Ah, yeah, eth_getLogs could use a lot of memory if you have a wide fromBlock and toBlock range. How many blocks are you requesting in these eth_getLogs calls? Do you have an example request that could crash ganache you could share?

[kifah1989]

i'm not specifying fromBlock and toBlock i'm using ethersjs to wait for a transaction logs and events. whoever, i'm now running the previous command with --detach flag and seems to be running smoothly for now.

[davidmurdoch]

Hm, detach mode shouldn't effect the memory at all. I'd love to look at a reproduction, if you are able to find a way to reliably reproduce the issue.