search

regular / unbzip2-stream

streaming unbzip2 implementatio in pure javascript for node and browsers
Other
29 stars 23 forks source link

JavaScript heap out of memory #17

Open braco opened 6 years ago

braco commented 6 years ago

Anyone else encounter this?

<--- Last few GCs --->

[35779:0x103001800]   204288 ms: Mark-sweep 1372.5 (1449.5) -> 1372.5 (1449.5) MB, 1296.8 / 0.0 ms  allocation failure GC in old space requested
[35779:0x103001800]   205584 ms: Mark-sweep 1372.5 (1449.5) -> 1372.5 (1418.0) MB, 1296.0 / 0.0 ms  last resort GC in old space requested
[35779:0x103001800]   206876 ms: Mark-sweep 1372.5 (1418.0) -> 1372.5 (1418.0) MB, 1292.4 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2223aa7255e9 <JSObject>
    1: f [/private/var/www/data/node_modules/unbzip2-stream/index.js:~25] [pc=0x3638d944dae9](this=0x2223759866f1 <JSGlobal Object>,b=97)
    2: /* anonymous */ [/private/var/www/data/node_modules/unbzip2-stream/lib/bzip2.js:~163] [pc=0x3638d945853e](this=0x222364102249 <Object map = 0x222302a76661>,bits=0x2223b85b5889 <JSFunction f (sfi = 0x2223d86f2a51)>,stream=0x2223fe782441 <JSFunction f (sf...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 2: node::FatalTryCatch::~FatalTryCatch() [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 4: v8::internal::Factory::NewUninitializedFixedArray(int) [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 5: v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::FastPackedSmiElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)0> >::GrowCapacity(v8::internal::Handle<v8::internal::JSObject>, unsigned int) [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 6: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/c/.nvm/versions/node/v9.11.1/bin/node]
 7: 0x3638d92842fd
Abort trap: 6
yanosh-igor commented 6 years ago

I have a similar issue. Looks like this module doesn't properly implement backpressure.

regular commented 6 years ago

Yes, lack of implementing back pressure could very well be the reason. If you write fast to the stream and consume the decompressed data slowly, decompressAndQueue will quickly build up a buffer queue of decompressed data. There probably should be a high watermark that causes the stream to pause. I don't actually use this module, so if one of you feels inclined to implement and test this, I'll certainly be willing to merge a pull request.