The CLI build fails when bundling big files

[ritave]

The mm-snap build command fails when big files are included in the bundle (for example 100mb).

The bug occurred in following scenarios

• Reading big files using fs.readFileSync and brfs Browserify plugin
• Building a snap with a large WASM (hex-encoded) dependency

An example stack trace of the error:

$ mm-snap build
Write error: Maximum call stack size exceeded
RangeError: Maximum call stack size exceeded
    at RegExp.exec (<anonymous>)
    at scan (/Users/redacted/src/cabal/snap/node_modules/@chainsafe/strip-comments/lib/parse.js:49:25)
    at parse (/Users/redacted/src/cabal/snap/node_modules/@chainsafe/strip-comments/lib/parse.js:90:20)
    at Object.module.exports [as default] (/Users/redacted/src/cabal/snap/node_modules/@chainsafe/strip-comments/index.js:35:18)
    at postProcess (/Users/redacted/src/cabal/snap/node_modules/@metamask/snaps-cli/dist/cmds/build/utils.js:64:51)
    at Object.writeBundleFile (/Users/redacted/src/cabal/snap/node_modules/@metamask/snaps-cli/dist/cmds/build/utils.js:31:45)
    at /Users/redacted/src/cabal/snap/node_modules/@metamask/snaps-cli/dist/cmds/build/bundle.js:47:75
    at /Users/redacted/src/cabal/snap/node_modules/browserify/index.js:843:13
    at ConcatStream.<anonymous> (/Users/redacted/src/cabal/snap/node_modules/browserify/node_modules/concat-stream/index.js:37:43)
    at ConcatStream.emit (node:events:532:35)
Write error: Failed to unlink mangled file.

CC @lsankar4033 and @piotr-roslaniec

[rekmarks]

The above stack trace indicates that the maximum call stack is exceeded by the RegExp.exec call in @chainsafe/strip-comments. Do you get the same error in both scenarios you described?

[piotr-roslaniec]

I think this particular stack trace was produced by @lsankar4033, but I can confirm it looks the same on my end. I can confirm that it's caused by ingestion of a large file - it doesn't really matter what the contents are, it's an issue with a build pipeline.

The error is reproduced here.

My stack trace:

$ mm-snap build
[BABEL] Note: The code generator has deoptimised the styling of <redacted>/regexp-error-snap/src/large-file.js as it exceeds the max of 500KB.
<redacted>/regexp-error-snap/node_modules/@chainsafe/strip-comments/lib/parse.js:49
    const match = regex.exec(remaining);
                        ^

RangeError: Maximum call stack size exceeded
    at RegExp.exec (<anonymous>)
    at scan (<redacted>/regexp-error-snap/node_modules/@chainsafe/strip-comments/lib/parse.js:49:25)
    at parse (<redacted>/regexp-error-snap/node_modules/@chainsafe/strip-comments/lib/parse.js:90:20)
    at module.exports (<redacted>/regexp-error-snap/node_modules/@chainsafe/strip-comments/index.js:35:18)
    at postProcessBundle (<redacted>/regexp-error-snap/node_modules/@metamask/snap-utils/dist/post-process.js:31:56)
    at SnapsBrowserifyTransform._flush (<redacted>/regexp-error-snap/node_modules/@metamask/snaps-browserify-plugin/dist/plugin.js:40:68)
    at SnapsBrowserifyTransform.prefinish (internal/streams/transform.js:147:10)
    at SnapsBrowserifyTransform.emit (events.js:400:28)
    at prefinish (internal/streams/writable.js:630:14)
    at finishMaybe (internal/streams/writable.js:638:5)

[rekmarks]

Does it blow up at build time with --strip-comments false?

It will probably blow up during the eval step if that's turned off, so the whole command should succeed with the above flag and --eval false.

[rekmarks]

A temporary workaround would be to stringify the WebAssembly file and inject it in a post-processing step that runs after mm-snap build --eval false, a trick which I believe some of you may already have employed. We can fix this on our end, but not until the next release of this monorepo (hopefully later this week).

For maintainers, we'd have to modify our implementation of user-specified bundler transforms such that users can determine the order in which plugins are applied. Right now, there's no way for a user to create a Browserify plugin or transform that in-lines the WebAssembly contents after our plugin is run. (Or, there probably is, but it would be hacky.)

I think we could do this in a non-breaking way by somehow modifying these lines such that users can add plugins both before and after our plugin is applied. Other suggestions are also welcome.

[ritave]

@rekmarks Probably the first thing we should do is to fix the strip-comments implementation as not to be able to reach the stack limit. I imagine it's because of lookbehind/ahead in the regex. Also move to a lazy streaming parser inside strip-comments

[FrederikBolding]

I agree with Olaf. I'll take another look at the implementation today.