OOM errors

[shark0der]

Description

I am using multiple compiler versions and I started to get OOM errors. The compilation itself works but the OOM error happens after recompiling a single contract and running the tests. After force recompiling - everything works again. The error is tricky to reproduce and I don't yet have a certain way to do it.

Hinted by @alcuadrado I tried passing the --max-memory argument which seems to set the --max-old-space-size node option but it doesn't work! I've tried using npx hardhat --max-memory 8192 test and node_modules/.bin/hardhat --max-memory 8192 test, in both cases it crashes. I was able to get it working by setting the environment variable NODE_OPTIONS="--max-old-space-size=4096".

The crash itself seems to happen while running the tests. I've put a few console.logs (the stack traces are useless). It reaches the console.log that's outside the describe (top-level), reaches the callback passed to describe but fails to reach the beforeEach and it callbacks.

The contract that I'm editing has pragma solidity ^0.5.0; and is compiled with 0.5.17. I have not encountered this before hardhat 2.0.7 (but this could totally be a coincidence!).

System info

$ node -v
v12.19.0

$ npm -v
6.14.11

$ npm ls --depth 0
repo@1.0.0 /w/contracts-dev
├── @nomiclabs/hardhat-truffle5@2.0.0
├── @nomiclabs/hardhat-web3@2.0.0
├── @openzeppelin/contract-loader@0.6.1
├── @openzeppelin/contracts@2.5.1
├── @openzeppelin/test-helpers@0.5.9
├── @truffle/hdwallet-provider@1.0.42
├── @typechain/truffle-v5@3.0.0
├── @uniswap/v2-core@1.0.1
├── @uniswap/v2-periphery@1.1.0-beta.0
├── bn.js@5.1.2
├── chai@4.2.0
├── decimal.js@10.2.1
├── dotenv@8.2.0
├── eslint@6.8.0
├── eslint-config-semistandard@15.0.1
├── eslint-config-standard@14.1.1
├── eslint-plugin-import@2.22.0
├── eslint-plugin-node@11.1.0
├── eslint-plugin-promise@4.2.1
├── eslint-plugin-standard@4.0.1
├── ethlint@1.2.5
├── hardhat@2.0.7
├── mocha@7.2.0
├── nodemon@2.0.4
├── UNMET PEER DEPENDENCY truffle@^5.0.0
├── truffle-flattener@1.4.4 (github:shark0der/truffle-flattener#3f9f5d9152b122eb2862fe94500c47539b8fd672)
├── ts-generator@0.1.1
├── typechain@4.0.0
└── web3@1.3.0

npm ERR! peer dep missing: truffle@^5.0.0, required by @typechain/truffle-v5@3.0.0
npm ERR! peer dep missing: bn.js@^4.11.0, required by chai-bn@0.2.1
npm ERR! peer dep missing: @babel/core@^7.0.0-0, required by @babel/plugin-transform-runtime@7.11.0
npm ERR! peer dep missing: solium@^1.0.0, required by solium-plugin-security@0.1.1
npm ERR! missing: glob@7.1.2, required by mocha@4.1.0
npm ERR! missing: glob@7.1.6, required by ts-generator@0.1.1
npm ERR! peer dep missing: typescript@>=3.7.0, required by ts-essentials@6.0.7

Hardhat config

require('dotenv').config();
require('@nomiclabs/hardhat-web3');
require('@nomiclabs/hardhat-truffle5');

const { task } = require('hardhat/config');
const ether = n => `${n}${'0'.repeat(18)}`;

task('test', async (_, hre, runSuper) => {
  hre.accounts = await hre.web3.eth.getAccounts();
  const testFiles = _.testFiles.length ? _.testFiles : ['./test/index.js'];
  await runSuper({ testFiles });
});

task('typechain', async (_, { config }) => {

  const { tsGenerator } = require('ts-generator');
  const { TypeChain } = require('typechain/dist/TypeChain');

  const cwd = process.cwd();
  const rawConfig = {
    files: `${config.paths.artifacts}/!(build-info|hardhat)/**/+([a-zA-Z0-9]).json`,
    outDir: 'types',
    target: 'truffle-v5',
  };

  await tsGenerator({ cwd }, new TypeChain({ cwd, rawConfig }));
});

const networks = {
  hardhat: {
    accounts: {
      count: 100,
      accountsBalance: ether(10000000),
    },
    allowUnlimitedContractSize: true,
    blockGasLimit: 12e6,
    gas: 12e6,
  },
  localhost: {
    blockGasLimit: 12e6,
    gas: 12e6,
  },
};

if (process.env.TEST_ENV_FORK) {
  networks.hardhat.forking = { url: process.env.TEST_ENV_FORK };
}

for (const network of ['MAINNET', 'KOVAN']) {
  const url = process.env[`${network}_PROVIDER_URL`];
  const accounts = [process.env[`${network}_ACCOUNT_KEY`]];
  networks[network.toLowerCase()] = { accounts, url };
}

const compilerSettings = process.env.ENABLE_OPTIMIZER
  ? { optimizer: { enabled: true, runs: 200 } }
  : {};

module.exports = {

  mocha: {
    exit: true,
    bail: true,
    recursive: false,
  },

  networks,

  solidity: {
    compilers: [
      { version: '0.5.17' }, // main contracts
      { version: '0.5.16' }, // uniswap v2 core
      { version: '0.6.6' }, // uniswap v2 peripherals
    ].map(compiler => ({ ...compiler, settings: compilerSettings })),
    overrides: {
      'contracts/modules/governance/Governance.sol': {
        version: '0.5.7',
        settings: compilerSettings,
      },
    },
  },
};

Error

$ node_modules/.bin/hardhat --max-memory 8192 test test/integration/index.js --show-stack-traces
test file required
describe() callback called

  INTEGRATION TESTS

<--- Last few GCs --->

[54356:0x46d2f20]    15856 ms: Scavenge 2041.1 (2047.8) -> 2041.3 (2048.5) MB, 10.8 / 0.0 ms  (average mu = 0.104, current mu = 0.047) allocation failure 
[54356:0x46d2f20]    15865 ms: Scavenge 2041.7 (2048.5) -> 2041.6 (2054.3) MB, 8.2 / 0.0 ms  (average mu = 0.104, current mu = 0.047) allocation failure 
[54356:0x46d2f20]    16117 ms: Scavenge 2045.4 (2054.8) -> 2045.8 (2051.5) MB, 10.5 / 0.0 ms  (average mu = 0.104, current mu = 0.047) allocation failure 

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1409219]
    1: StubFrame [pc: 0x138ff81]
Security context: 0x10c4cb7408d1 <JSObject>
    2: add [0x2db53a2a11e1] [/w/contracts-dev/node_modules/hardhat/internal/hardhat-network/stack-traces/contracts-identifier.js:~25] [pc=0xeb4c51c5b41](this=0x3f6c5e5f0d01 <BytecodeTrie map = 0x29a0a85773e9>,0x1fae9bab4c21 <Bytecode map = 0x29a0a8578929>)
    3: addBytecode [0x2db53a2a1101] [/w/contracts-dev/node_modules/...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa17c40 node::Abort() [node]
 2: 0xa1804c node::OnFatalError(char const*, char const*) [node]
 3: 0xb95a7e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb95df9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd53075  [node]
 6: 0xd53706 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd5ffc5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd60e75 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd6392c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd2a34b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x106c91e v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x1409219  [node]
[1]    54356 abort (core dumped)  node_modules/.bin/hardhat --max-memory 8192 test test/integration/index.js 

To summarize, there are 2 issues:

1. --max-memory flag does not work
2. The artifacts and/or the cache seem to get corrupted and/or cause high memory usage when a single contract is recompiled.

[nishuzumi]

same issue

[msfeldstein]

Here's a simple contract that reproes this on a brand new hardhat project https://gist.github.com/msfeldstein/4519c50b5e6b3be902d55868a6a73bda

[alcuadrado]

Hey, @msfeldstein, can you try again with the latest version of hardhat? We changed a few things that should fix this.

[fvictorio]

Closing in favor of https://github.com/NomicFoundation/hardhat/issues/3458.