search

enjalot / blockbuilder-search-index

download and process d3.js blocks for further indexing and visualization
BSD 3-Clause "New" or "Revised" License
24 stars 5 forks source link

js heap OOM with 8gb ram specified #60

Open micahstubbs opened 6 years ago

micahstubbs commented 6 years ago

so I'm updating my local elasticsearch, and I use this command from our docs

coffee --nodejs --max-old-space-size=8000 elasticsearch.coffee

oh no, at gist 30179 I see this error message 

indexed 30182 7067e1cc1b623959eacda6e34a2f63da
indexed 30181 7acb36eccb6280d95634f3d6f4d8f0f7
indexed 30179 c2acadc0809fcad97e403212333234d8

<--- Last few GCs --->

[12055:0x102801e00]   210399 ms: Mark-sweep 7845.3 (8060.4) -> 7844.9 (8060.9) MB, 247.6 / 0.0 ms  allocation failure GC in old space requested
[12055:0x102801e00]   210742 ms: Mark-sweep 7844.9 (8060.9) -> 7844.9 (8048.9) MB, 343.3 / 0.0 ms  last resort GC in old space requested
[12055:0x102801e00]   210957 ms: Mark-sweep 7844.9 (8048.9) -> 7844.9 (8043.9) MB, 215.4 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x184b31ca55e9 <JSObject>
    1: toString [buffer.js:~634] [pc=0x30f302b1d0b8](this=0x184b9b53c4c1 <Uint8Array map = 0x184b022da259>,encoding=0x184bef8022d1 <undefined>,start=0x184bef8022d1 <undefined>,end=0x184bef8022d1 <undefined>)
    2: arguments adaptor frame: 0->3
    3: /* anonymous */ [/Users/m/workspace/blockbuilder-search-index/elasticsearch.coffee:98] [bytecode=0x184b2ebd2fa1 offset=19](this=0x184bead866f1 <JS...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/m/.nvm/versions/node/v9.11.1/bin/node]
 2: node::FatalTryCatch::~FatalTryCatch() [/Users/m/.nvm/versions/node/v9.11.1/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/m/.nvm/versions/node/v9.11.1/bin/node]
 4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/Users/m/.nvm/versions/node/v9.11.1/bin/node]
 5: v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const>, v8::internal::PretenureFlag) [/Users/m/.nvm/versions/node/v9.11.1/bin/node]
 6: v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [/Users/m/.nvm/versions/node/v9.11.1/bin/node]
 7: node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) [/Users/m/.nvm/versions/node/v9.11.1/bin/node]
 8: void node::Buffer::(anonymous namespace)::StringSlice<(node::encoding)1>(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/m/.nvm/versions/node/v9.11.1/bin/node]
 9: 0x30f302248327
10: 0x30f302b1d0b8
➜  blockbuilder-search-index git:(micah/55/exp/parse-modules) ✗
micahstubbs commented 6 years ago

I'm on a machine with 16gb of physical memory, so I'm going to try this command with ~12gb and see if it works better:

coffee --nodejs --max-old-space-size=12000 elasticsearch.coffee

micahstubbs commented 6 years ago

ok,

coffee --nodejs --max-old-space-size=12000 elasticsearch.coffee

worked

indexed 37499 260ecc417a97e2af130cf8d89e9a24b4
indexed 37500 0c608d4724d2335f89f1a06f762e5313
indexed 37508 4d3450c295f1342f63b6bef88230e855
indexed 37505 2e368d82cc88fcc386f9f1792e20c69b
indexed 37504 604b907cdb944368a3d635d4f6663c6d
indexed 37503 db1ac0efe50844239f20aa3762dd1729
indexed 37507 35cc78e983f14aee9dc2ac571e6121ea
indexed 37506 c92fa45bace8fc78be67c76d90e20979
indexed 37501 44d27aa57120d799cc3f214976f716a2
indexed 37502 f850022a5309d979ac8e99fa08b6d4aa
done
skipped 0 missing files
micahstubbs commented 6 years ago

will update the docs