JavaScript heap out of memory

swernerx commented 7 years ago

I am using this in a medium large React project as part of the Webpack flow. For compression we are using the Babili-Webpack-Plugin. As this is just a small wrapper around Babili I figure the following issue is more related to Babili itself:


<--- Last few GCs --->

  344958 ms: Mark-sweep 1324.0 (1434.9) -> 1324.0 (1434.9) MB, 1946.1 / 0.0 ms [allocation failure] [GC in old space requested].
  346872 ms: Mark-sweep 1324.0 (1434.9) -> 1324.0 (1434.9) MB, 1913.9 / 0.0 ms [allocation failure] [GC in old space requested].
  348780 ms: Mark-sweep 1324.0 (1434.9) -> 1327.8 (1405.9) MB, 1907.5 / 0.0 ms [last resort gc].
  350727 ms: Mark-sweep 1327.8 (1405.9) -> 1331.7 (1405.9) MB, 1946.7 / 0.0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x381e5ffcfb51 <JS Object>
    1: visitQueue [/workspace/1_ui/ui-foundation/node_modules/babel-traverse/lib/context.js:~114] [pc=0x35f210c516c7] (this=0x2bf0d5c576b9 <a TraversalContext with map 0x22808a4e239>,queue=0x2bf0d5c57701 <JS Array[1]>)
    2: node [/workspace/1_ui/ui-foundation/node_modules/babel-traverse/lib/index.js:~94] [pc=0x35f210cf29bc] (...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [/usr/local/bin/node]
 4: v8::Utils::ApiCheck(bool, char const*, char const*) [/usr/local/bin/node]
 5: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 6: v8::internal::Factory::NewByteArray(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 7: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, bool, int, bool) [/usr/local/bin/node]
 8: v8::internal::PropertyAccessCompiler::GetCodeWithFlags(unsigned int, char const*) [/usr/local/bin/node]
 9: v8::internal::PropertyHandlerCompiler::GetCode(v8::internal::Code::Kind, v8::internal::Code::StubType, v8::internal::Handle<v8::internal::Name>) [/usr/local/bin/node]
10: v8::internal::NamedLoadHandlerCompiler::CompileLoadNonexistent(v8::internal::Handle<v8::internal::Name>) [/usr/local/bin/node]
11: v8::internal::NamedLoadHandlerCompiler::ComputeLoadNonexistent(v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>) [/usr/local/bin/node]
12: v8::internal::LoadIC::UpdateCaches(v8::internal::LookupIterator*) [/usr/local/bin/node]
13: v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>) [/usr/local/bin/node]
14: v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
15: 0x35f20b8092a7
Abort trap: 6

These are the relevant software versions:

Mac OS Sierra
Node 6.9.2
Babili Preset 0.0.9
Babel Core 6.18.2
Babel Traverse 6.19.0

Is there anything I can do to debug this anymore?

swernerx commented 7 years ago

I just played a little with the custom configuration option for using a alternative preset instead of of the default one.

When disabling all processing I get a few of these messages:

[BABEL] Note: The code generator has deoptimised the styling of "unknown" as it exceeds the max of "100KB".

I figure generally we might have a problem at working at a chunk level for babili as the chunks are generally quite massive.

swernerx commented 7 years ago

It seems to work with this setup:

new BabiliPlugin({
  comments: false,
  babili: {
    plugins: [
      "minify-dead-code-elimination",
      "minify-mangle-names",
      "minify-simplify"
    ]
  }
})

boopathi commented 7 years ago

What's the file size which when crossed gives this error - heap out of memory ?

Or is it happening only for your project for big file sizes ?

swernerx commented 7 years ago

I am not actually sure which chunk is producing this issue.

swernerx commented 7 years ago

With the limited set of plugins it's still using a ton of memory: 1.5GB highest - but it works (it does not do the minification thing for all files it seems - maybe that's related to the 100KB warning).

swernerx commented 7 years ago

BTW: Using a GB+ range of memory destroys the relevance for using this setup on any kind of CI server. These have typically somewhat limited memory/cpu offerings.

swernerx commented 7 years ago

Okay, when placing a "minified" inside the config this message is gone:

[BABEL] Note: The code generator has deoptimised the styling of "unknown" as it exceeds the max of "100KB".

Current config:

new BabiliPlugin({
  comments: false,
  babili: {
    minified: true,
    comments: false,
    plugins: [
      "minify-dead-code-elimination",
      "minify-mangle-names",
      "minify-simplify"
    ]
  }
})

osidenate commented 7 years ago

I get the same Out of Memory issue when running babili using BabiliPlugin/Webpack on my medium size React project. It works fine on my small React project.

amarant commented 7 years ago

I had the same problem, I used : node --max-old-space-size=4076 ... and it finished.

kangax commented 7 years ago

Similar report from @qfox "tried babili on the build and it just keeps on generating stuff. i killed it after 35mb or so." — https://gist.github.com/qfox/7307e414c5414b26b4f72b2b77b1ee30

derekdon commented 7 years ago

Same issues here on a large project.

new BabiliPlugin({
                test: /\.js($|\?)/i,
                comments: false,
                sourceMap: true
            }),

Tried @swernerx settings but the same error occurs.

new BabiliPlugin({
                comments: false,
                babili: {
                    minified: true,
                    comments: false,
                    plugins: [
                        "minify-dead-code-elimination",
                        "minify-mangle-names",
                        "minify-simplify"
                    ]
                }
            })

postspectacular commented 7 years ago

FWIW I noticed that babili goes into an infinite loop unless explicitly locally installed into a project. I.e. Installing globally and then using version via npm link babili will trigger the loop and eat up more & more memory, even with the smallest source files...

pvdz commented 7 years ago

@postspectacular fwiw I don't have it installed globally. There's very little that should need to be installed globally.

derekdon commented 7 years ago

@postspectacular same as @qfox, not installed globally.

joseSantacruz commented 7 years ago

same issue here

Windstalker commented 7 years ago

The issue was solved for me by turning off the option mangle. Please, fix this issue.

"env": {
      "production": {
        "only": [
          "app",
          "webpack"
        ],
        "presets": [
          [
            "babili",
            {
              "mangle": false
            }
          ]
        ],
        ...
      }
   }

vigneshshanmugam commented 7 years ago

@Windstalker can you confirm the babili version?

Windstalker commented 7 years ago

@vigneshshanmugam babel-preset-babili@0.1.2

davisford commented 7 years ago

Can confirm -- turning off mangle at least successfully minifies it.

├─┬ babili-webpack-plugin@0.1.1
│ ├── babel-core@6.25.0 deduped
│ ├── babel-preset-babili@0.1.2
│ └── webpack-sources@0.2.3

boopathi commented 7 years ago

Interesting. Thanks for digging into the issue and finding out that mangler is causing this. I'll try to debug this sometime soon.

brandonmp commented 7 years ago

I had the same problem, I used : node --max-old-space-size=4076 ... and it finished.

A+, thx.

this worked for me as well (but boy does the build take awhile to finish)

yocontra commented 7 years ago

Still getting the same issue as of July/August - disabling mangle didn't help.

<--- Last few GCs --->

[24150:0x102801600]   126688 ms: Mark-sweep 1403.4 (1490.7) -> 1403.2 (1459.7) MB, 2206.9 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2207 ms) last resort
[24150:0x102801600]   128938 ms: Mark-sweep 1403.2 (1459.7) -> 1403.2 (1459.7) MB, 2249.9 / 0.0 ms  last resort

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xd18d03a66a1 <JS Object>
    1: getBindingIdentifiers [/Users/contra/Projects/staeco/node_modules/babili-webpack-plugin/node_modules/babel-types/lib/retrievers.js:~20] [pc=0x10c73d68fe4e](this=0x26c47a909239 <an Object with map 0x2405980dddd1>,node=0x2c5b756e2ad9 <a Node with map 0x52ecb9d1f61>,duplicates=0x3363d09023b1 <true>,outerOnly=0x3363d0902311 <undefined>)
    2: arguments adaptor frame: 2->3
    3: registerBind...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
 4: v8::internal::Factory::NewTransitionArray(int) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
 5: v8::internal::TransitionArray::Insert(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::SimpleTransitionFlag) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
 6: v8::internal::Map::CopyReplaceDescriptors(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Handle<v8::internal::LayoutDescriptor>, v8::internal::TransitionFlag, v8::internal::MaybeHandle<v8::internal::Name>, char const*, v8::internal::SimpleTransitionFlag) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
 7: v8::internal::Map::CopyAddDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
 8: v8::internal::Map::CopyWithField(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::FieldType>, v8::internal::PropertyAttributes, v8::internal::Representation, v8::internal::TransitionFlag) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
 9: v8::internal::Map::TransitionToDataProperty(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
10: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
11: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
12: v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
13: v8::internal::Runtime_SetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/contra/.nvm/versions/node/v7.10.0/bin/node]
14: 0x10c73c5043a7

iamacup commented 7 years ago

Hey team

I am experiencing this issue - but i have a working before and after with a specific library if it helps you debug:

https://github.com/iamacup/sliips-ui/commit/c437975c837a1b05476f3c36f716e59c5f4a756b

This commit works fine - if you do npm install then yarn build:local you will see it takes awhile to get through 91% asset optimisation, but it completes after a couple of minutes.

https://github.com/iamacup/sliips-ui/commit/c1d28a946d2e868baca7bae9c656339bd4744fb3

This commit is totally broken and yarn build:local will hang on the 91% for an hour or so and then run out of heap - even at 16 GB.

The only difference between the two commits is that i added a 571 KB dependency to the vendors.js file - specifically the pdf.js project's pdf.js file pdfjs-dist/build/pdf.js

I am wondering if its something to do with the fact that pdf.js contains webpack markup that is confusing things? anyway I will leave it with you - hope this helps.

tkharuk commented 6 years ago

Hi there,
not sure what is the real root cause for this but had the same issue which was a total blocker for CI. Neither allocating more memory nor turning off some flags didn't help us much and it was either OOM or build would take over an hour.
Our project is probably on a medium scale with minified bundle weighing somewhere near 4MB.

In the end had to switch to UglifyJS. It is a pity that such problem occurs recurrently and it is not clear at first glance what is the true root cause.

node: 7.9
npm: 5.0
used: react, mobx, decorators, babel-polyfill - targeting IE10+

awkaiser commented 6 years ago

I also ran into this problem on a larger React project and ended up moving from babel-minify-webpack-plugin (791 seconds before running out of memory) to uglifyjs-webpack-plugin (85 seconds, completed minification with source maps).

damianobarbati commented 6 years ago

@awkaiser what about bundle size? Did you compare them after moving to uglify? I actually didn't manage to get it working, weird errors and weird options (I'm talking about parse/output options that makes me think uglify attempt to trasform again es5<=>es8 code which is already done by babel). On the otherside babel-minify is flawless to me, just... slow :P

awkaiser commented 6 years ago

@damianobarbati Since the Uglify output size was roughly within expectations, I did not bother noting a size comparison with Babel Minify (which wasn't completing its build at all; would have had to roll back to an earlier version of the project).

tkharuk commented 6 years ago

@damianobarbati @awkaiser as for me, I don't recall exact numbers, but the results were roughly the same.

caesarsol commented 6 years ago

Same problem here, with multiple projects... Seems to work well until a certain size limit is reached, sometimes the addition of a dependency. Is that your case @iamacup ?

Is there some investigation going on this? Do you need any insight?

Thanks, hooray for Babel team! ;)

taoabc commented 6 years ago

Mark, same issue here. I use babel-minify-webpack-plugin to minify pdfjs-dist/build/pdf.worker.js

caesarsol commented 6 years ago

@taoabc we too have pdfjs in our bundle! seems it contains a very high number of identifiers.

damianobarbati commented 6 years ago

@taoabc @caesarsol I'm having similar problem with js-xlsx (~400kb file https://github.com/SheetJS/js-xlsx/blob/master/dist/xlsx.min.js)!

You can mitigate the memory/heap issue firing the webpack executable manually like this:

node --max_old_space_size=8192 ./node_modules/webpack/bin/webpack.js --config ./config/webpack.config.js

The real problem is CI services like travis are hanging anyway when the build takes more than 10mins (travis_wait does not work). Hoping for this pull request to be released soon to exclude it from the minification https://github.com/webpack-contrib/babel-minify-webpack-plugin/pull/75 🤞🏻

iamacup commented 6 years ago

@caesarsol @taoabc we had this problem with pdfjs as well - ended up including it via a CDN seperately instead as it caused unacceptable build time

Telokis commented 6 years ago

Still happening nowadays.

<--- Last few GCs --->

[256:0x5616f605c000]   321540 ms: Mark-sweep 1299.0 (1482.1) -> 1299.0 (1482.6) MB, 3589.2 / 0.0 ms  allocation failure GC in old space requested
[256:0x5616f605c000]   324694 ms: Mark-sweep 1299.0 (1482.6) -> 1299.0 (1442.6) MB, 3152.9 / 0.0 ms  last resort GC in old space requested
[256:0x5616f605c000]   327849 ms: Mark-sweep 1299.0 (1442.6) -> 1299.0 (1440.1) MB, 3154.7 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x38903bb25501 <JSObject>
    1: add(this=0x3dd9de8c2299 <Set map = 0x191511884cf9>,0x282971196429 <Object map = 0x3a45224f4c99>)
    2: rename [/builds/Telokis/Nyzia/Client/node_modules/babel-plugin-minify-mangle-names/lib/index.js:~490] [pc=0x18476e91c061](this=0x1c1926c93581 <Mangler map = 0x31d03d945909>,scope=0x4de54ec2cd9 <Scope map = 0x3a45224f7b59>,binding=0x4de54ec9439 <Binding map = 0x3a45224f7109>,oldName=0x18...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

jakobfdev commented 6 years ago

Same problem here with an electron-webpack app, also using xlsx as mentioned before. Setting --max_old_space_size=8192 also didn't help.

EDIT:

 95% [1] additional asset processing                                                        
<--- Last few GCs --->

[19886:0x102801e00]   349054 ms: Mark-sweep 1399.6 (1578.4) -> 1399.6 (1578.4) MB, 2444.2 / 0.0 ms  allocation failure GC in old space requested
[19886:0x102801e00]   351345 ms: Mark-sweep 1399.6 (1578.4) -> 1399.6 (1547.4) MB, 2290.2 / 0.0 ms  last resort GC in old space requested
[19886:0x102801e00]   353727 ms: Mark-sweep 1399.6 (1547.4) -> 1399.6 (1547.4) MB, 2381.2 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x33a4d9025ee1 <JSObject>
    1: SequenceExpression [/Volumes/www/wkn-toolbox/node_modules/babel-generator/lib/generators/expressions.js:~91] [pc=0x233c7107a669](this=0x33a41b1f0ff9 <Generator map = 0x33a4b0f82bf9>,node=0x33a48d85f1d1 <Object map = 0x33a49e868c71>)
    2: arguments adaptor frame: 2->1
    3: print [/Volumes/www/wkn-toolbox/node_modules/babel-generator/lib/printer.js:~267] [pc=0x233c714668e4](this=0x33a41...

ghost commented 6 years ago

Just to add some information. I have been facing same issue. Interestingly we find one mistake we made in our code base. We have been using 'babel-register' in index.js which is responsible for dynamic loadings of file. By mistake we are trying to load compiled main.js bundle file using dynamic module. So, babel is parsing it again, finally running out of memory. Please make sure not to parse compiled version by babel. now it is working for us!

Roman86 commented 5 years ago

I hope it helps (at least worked in my case): Try to increase the node memory limit (which is 1.4 GB by default) by adding corresponding option to your node command: --max-old-space-size=4076 or even 8192. Original solution source: https://stackoverflow.com/questions/34356012/how-to-increase-nodejs-default-memory

UPD: I've found that this advice was already given here in 2017. So it still helps in 2019 =)

mikeRChambers610 commented 5 years ago

@roman86 how do you run the node command? I see so many responses saying "use" --max-old-space-size=4076 but never seeing any instruction on how to "use" it. Would you mind providing me the direction please? Thanks!!

torson commented 4 years ago

@mikeRChambers610 you need to call brunch using node executable. So if you previously had a task running brunch executable directly :

            'shell:brunch': {
                command: 'brunch build'
            },

now set it like this to increase the heap size limit:

            'shell:brunch': {
                command: 'node --max-old-space-size=1024 `which brunch` build -p'
            },

which brunch is there so you don't need to set absolute path of the file, so it works in all environments.

jackmahoney commented 3 years ago

This is a major issue for me. I have a project with around 1200 pages and the memory needs keep growing. I can't build in CI anymore and have to run it locally on a powerful machine with --max-old-space-size=8192. Building also takes about 5 minutes. I really like Vue and Vuepress but I wish I'd know about this issue before I invested so heavily. With Babel, Vue, Remark and all the tech all being called to compile the project (in parallel I assume) I wouldn't know where to begin in diagnosing the root of this issue. Perhaps I should use a cache flag and save the cache to version control? I don't think continuously increasing the memory limits is a viable long term solution.

babel / minify

JavaScript heap out of memory #332