Forked process with "import" (ES6) fails in Heroku env with 512mb of memory

[markb-trustifi]

Version

16.13.2

Platform

Darwin MacBook-Pro.local 20.6.0 Darwin Kernel Version 20.6.0: Wed Nov 10 22:23:07 PST 2021; root:xnu-7195.141.14~1/RELEASE_X86_64 x86_64

Subsystem

No response

What steps will reproduce the bug?

Parent process:

import {fork} from 'node:child_process';
let options = {
   stdio: [ 'pipe', 'pipe', 'pipe', 'ipc' ],
   env: Object.assign({}, process.env, { NODE_DEBUG: '*' })
};
const child = fork('./childFork.js',['param'], options);
child.on('message', message => {
   console.info('message from child:' + message);
})

Child forked process:

process.on('uncaughtException', function (err) {
   process.send({stack: err.stack, message: err.message}, null, {}, _ => {
      process.exit(1);
   });
});
process.send('READY');

When project's package.json is configured to use the new imports instead of the old requires ("type": "module") the forked process fails with SIGABRT error. It fails only on Heroku Standard X1 env (512Mb) and runs well on X2 env (1Gb). The main process runs with parameters: node --expose-gc --optimize_for_size --max_old_space_size=9200 main.js

How often does it reproduce? Is there a required condition?

The import forks fail on Heroku smallest 512Mb envs.

What is the expected behavior?

The child forks with imports should run on the same env as with requires.

What do you see instead?

The child forked process starts to load:

MODULE 193: Module._load REQUEST /app/.heroku/heroku-nodejs-plugin parent: internal/preload
MODULE 193: looking for "/app/.heroku/heroku-nodejs-plugin" in ["/app/node_modules","/node_modules","/app/.node_modules","/app/.node_libraries","/app/.heroku/node/lib/node"]
MODULE 193: load "/app/.heroku/heroku-nodejs-plugin/index.js" for module "/app/.heroku/heroku-nodejs-plugin/index.js"
MODULE 193: Module._load REQUEST util parent: /app/.heroku/heroku-nodejs-plugin/index.js
MODULE 193: load native module util
MODULE 193: Module._load REQUEST http parent: /app/.heroku/heroku-nodejs-plugin/index.js
MODULE 193: load native module http
MODULE 193: Module._load REQUEST https parent: /app/.heroku/heroku-nodejs-plugin/index.js
MODULE 193: load native module https
MODULE 193: Module._load REQUEST url parent: /app/.heroku/heroku-nodejs-plugin/index.js
MODULE 193: load native module url
MODULE 193: Module._load REQUEST events parent: /app/.heroku/heroku-nodejs-plugin/index.js
MODULE 193: load native module events
MODULE 193: Module._load REQUEST ./heroku-nodejs-plugin.node parent: /app/.heroku/heroku-nodejs-plugin/index.js
MODULE 193: RELATIVE: requested: ./heroku-nodejs-plugin.node from parent.id /app/.heroku/heroku-nodejs-plugin/index.js
MODULE 193: looking for ["/app/.heroku/heroku-nodejs-plugin"]
MODULE 193: load "/app/.heroku/heroku-nodejs-plugin/heroku-nodejs-plugin.node" for module "/app/.heroku/heroku-nodejs-plugin/heroku-nodejs-plugin.node"
TIMER 193: no 20000 list was found in insert, creating a new one

...and suddenly fails with SIGABRT.

When the main process runs on a bigger env with 1Gb of memory it loads child process successfully:

ESM 193: Storing file:///app/childFork.js in ModuleMap
ESM 193: Translating StandardModule file:///app/childFork.js
message from child: READY

When the main process runs with the old-fashion require config it runs forks well even on the 512Mb env:

MODULE 193: looking for "/app/childFork.js" in ["/app/.node_modules","/app/.node_libraries","/app/.heroku/node/lib/node"]
MODULE 193: load "/app/childFork.js" for module "."
message from child: READY

Additional information

Please confirm if import projects have higher memory requirements than the require projects.

[benjamingr]

@nodejs/modules

[guybedford]

This may also be related to https://github.com/nodejs/node/issues/40201 for the Wasm execution of cjs-module-lexer. One easy fix might be to switch to a JS implementation instead of Wasm, as es-module-lexer does have a JS alternative.

[GeoffreyBooth]

One easy fix might be to switch to a JS implementation instead of Wasm, as es-module-lexer does have a JS alternative.

Yes, but wouldn’t that entail a performance hit? I thought the reason es-module-lexer was written in Wasm was because it ran faster.

I feel like this is a naïve suggestion, but I don’t suppose we could be like “when available memory is below X, use the js es-module-lexer, else use the wasm es-module-lexer.” But then we’re shipping two versions and would need to ensure they stay in sync and always return the same results for the same input.

[guybedford]

The performance of the asm.js build of es-module-lexer turned out to be pretty good - https://github.com/guybedford/es-module-lexer/pull/86. A similar build for cjs-module-lexer could be possible.

[markb-trustifi]

I retested this issue now on Node.js versions 16.17 and 18.9 with latest Heroku stack 22 and it is still failing.

[bnoordhuis]

Apropos this:

node --expose-gc --optimize_for_size --max_old_space_size=9200 main.js

9200 MB doesn't make sense if the instance only has 512 MB available. What happens when you set it 256 or 128?