Javascript Heap Memory Overload when Building with Meteor

[Jdesk]

Description:

When building out locally with meteor I am running into a Javascript heap memory overload when trying to build out this package.

Here is the script I'm using to build this out:

set -x
set -euvo pipefail
IFS=$'\n\t'

# Requires Node.js version 4.x
# Do not run as root

DEPLOY_DIR=~/Desktop/convergent/build
TOOL_NODE_FLAGS=--max-old-space-size=4096

### BUILD
meteor npm install --save babel-runtime --save bcrypt
meteor npm run postinstall

# on the very first build, meteor build command should fail due to a bug on emojione package (related to phantomjs installation)
# the command below forces the error to happen before build command (not needed on subsequent builds)
set +e
meteor add rocketchat:lib
set -e

meteor build --server-only --directory $DEPLOY_DIR

### RUN
cd $DEPLOY_DIR/bundle/programs/server
npm install

cd $DEPLOY_DIR/bundle
NODE_ENV=production \
PORT=3000 \
ROOT_URL=http://localhost:3000 \
MONGO_URL=mongodb://localhost:27017/rocketchat \
MONGO_OPLOG_URL=mongodb://localhost:27017/local \
node main.js

Steps to reproduce:

1. Enter develop branch of rocket chat
2. Run build bash script

Expected behavior:

Meteor bundle created

Actual behavior:

Javascript heap memory overload

Server Setup Information:

• Version of Rocket.Chat Server: 0.69.1
• Operating System: osx high sierra
• Deployment Method: meteor local
• Number of Running Instances: local
• DB Replicaset Oplog:
• NodeJS Version: 8.11.4
• MongoDB Version: 4.0.1

Additional context

Relevant logs:

IFS=$'\n\t'
+ IFS='
    '

# Requires Node.js version 4.x
# Do not run as root

DEPLOY_DIR=~/Desktop/convergent/build
+ DEPLOY_DIR=/Users/jamesbarlow/Desktop/convergent/build
TOOL_NODE_FLAGS=--max-old-space-size=4096
+ TOOL_NODE_FLAGS=--max-old-space-size=4096

### BUILD
meteor npm install --save babel-runtime --save bcrypt
+ meteor npm install --save babel-runtime --save bcrypt
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but npm-shrinkwrap.json was generated for lockfileVersion@0. I'll try to do my best with it!

> bcrypt@2.0.1 install /Users/jamesbarlow/Desktop/convergent/rc2/node_modules/bcrypt
> node-pre-gyp install --fallback-to-build

[bcrypt] Success: "/Users/jamesbarlow/Desktop/convergent/rc2/node_modules/bcrypt/lib/binding/bcrypt_lib.node" is installed via remote
npm WARN chai-as-promised@6.0.0 requires a peer of chai@>= 2.1.2 < 4 but none is installed. You must install peer dependencies yourself.
npm WARN postcss-html@0.23.7 requires a peer of postcss-syntax@^0.10.0 but none is installed. You must install peer dependencies yourself.
npm WARN postcss-markdown@0.23.7 requires a peer of postcss-syntax@^0.10.0 but none is installed. You must install peer dependencies yourself.

+ bcrypt@2.0.1
+ babel-runtime@6.26.0
updated 2 packages and audited 13647 packages in 15.963s
found 119 vulnerabilities (20 low, 87 moderate, 12 high)
  run `npm audit fix` to fix them, or `npm audit` for details
meteor npm run postinstall
+ meteor npm run postinstall

> Rocket.Chat@0.69.0 postinstall /Users/jamesbarlow/Desktop/convergent/rc2
> cd packages/rocketchat-katex && npm i

audited 2 packages in 1.027s
found 0 vulnerabilities

# on the very first build, meteor build command should fail due to a bug on emojione package (related to phantomjs installation)
# the command below forces the error to happen before build command (not needed on subsequent builds)
set +e
+ set +e
meteor add rocketchat:lib
+ meteor add rocketchat:lib
rocketchat:lib without a version constraint has already been added.

set -e                                        
+ set -e

meteor build --server-only --directory $DEPLOY_DIR
+ meteor build --server-only --directory /Users/jamesbarlow/Desktop/convergent/build
up to date in 1.499se plugins                \
   Building for web.browser                  |
<--- Last few GCs --->

[39210:0x103800000]   255673 ms: Mark-sweep 1452.8 (1534.9) -> 1452.8 (1503.9) MB, 2310.3 / 0.1 ms  last resort GC in old space requested
[39210:0x103800000]   257984 ms: Mark-sweep 1452.8 (1503.9) -> 1452.8 (1503.9) MB, 2311.3 / 0.1 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x16fc4d3a5879 <JSObject>
    1: ReferencedIdentifier [/Users/jamesbarlow/.meteor/packages/ecmascript/.0.10.9.1xi3zkn.ato9m++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/@babel/traverse/lib/scope/index.js:~128] [pc=0x3e0d47ce8d1](this=0x16fc7c27fcd9 <Object map = 0x16fcb4fa20d1>,path=0x16fc5e03bcc1 <NodePath map = 0x16fcb4fa26a9>,state=0x16fc7c27fcd9 <Object ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
build.sh: line 22: 39210 Abort trap: 6           meteor build --server-only --directory $DEPLOY_DIR
<!-- Logs from both SERVER and BROWSER -->
<!-- For more information about collecting logs please see: https://rocket.chat/docs/contributing/reporting-issues#gathering-logs -->```

[geekgonecrazy]

How long after it starts there is it crashing? Also are you running this build script directly on osx? I'd recommend building on a Linux box. While yes some of us develop on osx our build scripts are really targeted for linux

[Jdesk]

Any system specs required as meteor seems to be real memory and cpu intensive. I am trying to build out directly on osx.

[geekgonecrazy]

It looks like you are setting the memory limit to 4gb. Do you actually have that much free?

On osx I think it does consume more memory when building. Linux is really your best bet when building

[Jdesk]

I am setting it to 4gb - definitely have it free. I'm going to try building out on linux and will see how that goes. When you guys build, what are the specs of your linux boxes? how much memory do you have?

[Jdesk]

I'm getting the same thing on linux - created an ec2 instance for this build. T2-Large with 8gb of ram and 4cpus.

Note, I've also removed the memory limit completely after setting it at 4gb the first run. Both builds resulted in the error below.

==== JS stack trace =========================================

Security context: 0x1fbe9f025879 <JSObject>
    1: ReferencedIdentifier [/home/ubuntu/.meteor/packages/ecmascript/.0.10.9.10owkt8.g8vj++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/@babel/traverse/lib/scope/index.js:~128] [pc=0x279f623546b1](this=0x1a718d60fd89 <Object map = 0x3d8afca9bf31>,path=0x19c2e93eb309 <NodePath map = 0x3d8afca9c509>,state=0x1a718d60fd89 <Object map = ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 2: 0x8c21ec [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 5: v8::internal::Factory::NewUninitializedFixedArray(int) [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 6: 0xd4b133 [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 7: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [/home/ubuntu/.meteor/packages/meteor-tool/.1.6.1_3.ms7bj2.ffl++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node]
 8: 0x279f61b842fd
build.sh: line 21: 13333 Aborted                 (core dumped) meteor build --server-only --directory $DEPLOY_DIR```

[YuhuaDeng]

Same issue here, I also built it on a CentOS 7 Box, and facing the same issue.

[Jdesk]

Just a bit of background on why we are trying to do this. We forked the repo and want to customize some things. I'm DevOps so I won't be doing the development. Our deploy method is docker, but this build is giving me a huge headache.

I've got this to build without using the --save-bcrypt flag (which I added as a recommendation from the logs saying that we are using a pure javascript implementation and this runs 3x slower. Try running npm install --save-bcrypt when building"

We have some customizations we want to implement. I'm not sure if adding to the codebase directly or creating applications that interact with the rocketchat api is a better solution. I'm leaning towards the latter. One example of something we are trying to accomplish is having the ability to create scheduled messages that will be broadcast to specific channels. Another example of something we want to do is the ability to audit specific users and be able to export their chats to csv.

[geekgonecrazy]

@rocketchat/core anyone have any ideas?

Regarding fork or use api's. We definitely prefer using integrations, apps, and our API.

The ability to schedule things with our apps framework is on the roadmap for fairly soon I believe. Once that's added you could actually just write an app to add that additional functionality. Idea being customize with out having to maintain a fork.

[Jdesk]

@Rocketchat/core just checking in to see if you guys have any ideas for me :-)

[sampaiodiego]

recently I had to use this env var to be able to build rocket.chat from within a VM:

export TOOL_NODE_FLAGS="--max_old_space_size=4096"

[Jdesk]

@sampaiodiego I actually tried the same thing - and it's still failing when I use the flag --save-bcrypt (which I need apparently to avoid using a pure javascript implementation which is 3x slower).

[geekgonecrazy]

@Jdesk hm.. I don't recall having any of our scripts using --save-bcrypt my understanding is when you do npm install after the bundle is produced it will install and compile bcrypt if it can.

So you should be good to just drop that instead of continuing to be stuck over that.

[localguru]

Same here, @sampaiodiego export TOOL_NODE_FLAGS="--max_old_space_size=4096"helped

[reetp]

Is this still an issue or can it be closed?

[sampaiodiego]

we might have a troubleshooting section on our README and add things like this.. wdyt?