chrisspiegl
commented
6 years ago As an addon, I'd like to provide the full error messages readout. I also attached the processed profile log to this comment.
It clearly shows that the issue seems to be inside the C++ section of the script.
My other assumptions are that it's because the script tries to read the full WAV file into memory instead of processing it in chungs. Maybe there is a way to make it process the buffer and not store all of it? Just the meta data? I am not that deep in it obviously, but I hope for a solution 🤞 .
Maybe it helps to better understand the issue at hand:
processed.txt: the profiling readout
% node index.js --max-old-space-size=14192 --optimize-for-size !1210
WaveFile processing started 2018-06-20T05:16:41.626Z
Loading WaveFile: ./test-cat-org.wav
Reading WaveFile
<--- Last few GCs --->
[13448:0x104000000] 443 ms: Mark-sweep 17.7 (23.1) -> 14.1 (21.4) MB, 0.7 / 0.0 ms (+ 4.3 ms in 6 steps since start of marking, biggest step 1.2 ms, walltime since start of marking 309 ms) finalize incremental marking via stack guard GC in old space[13448:0x104000000] 5027 ms: Mark-sweep 770.1 (777.4) -> 605.1 (612.4) MB, 174.2 / 0.0 ms (+ 18.0 ms in 4 steps since start of marking, biggest step 6.6 ms, walltime since start of marking 4489 ms) allocation failure GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x67fc5ca5ee1 <JSObject>
2: SimpleSlice(aka SimpleSlice) [native array.js:1] [bytecode=0x67f3437e591 offset=41](this=0x67f1c702311 <undefined>,p=0x67f67c02751 <Uint8Array map = 0x67f6d241d91>,O=0,P=490490104,Q=490490104,R=0x67f67c02731 <JSArray[490490104]>)
4: ArraySliceFallback [native array.js:1] [bytecode=0x67f3437dee9 offset=281](this=0x67f67c02751 <Uint8Array map = 0x67f6d241d91>,at=0x67f1c702311 <undefined>...
FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
1: node::Abort() [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
4: v8::internal::HashTable<v8::internal::SeededNumberDictionary, v8::internal::SeededNumberDictionaryShape>::EnsureCapacity(v8::internal::Handle<v8::internal::SeededNumberDictionary>, int, v8::internal::PretenureFlag) [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
5: v8::internal::Dictionary<v8::internal::SeededNumberDictionary, v8::internal::SeededNumberDictionaryShape>::Add(v8::internal::Handle<v8::internal::SeededNumberDictionary>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
6: v8::internal::(anonymous namespace)::DictionaryElementsAccessor::AddImpl(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, unsigned int) [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
7: v8::internal::JSObject::AddDataElement(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow) [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
8: v8::internal::JSObject::DefineOwnPropertyIgnoreAttributes(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::JSObject::AccessorInfoHandling) [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
9: v8::internal::JSObject::CreateDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::ShouldThrow) [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
10: v8::internal::Runtime_CreateDataProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/spieglio/.nvm/versions/node/v8.9.4/bin/node]
11: 0x2e89153c6838
12: 0x2e8915475209
[1] 13448 abort node index.js --max-old-space-size=14192 --optimize-for-size
rochars
Hi,
I finally found this library. I was looking for something like it for a long time. But sadly it appears to have pretty strong limitations (or I have not found the documentation for my problem):
In my scenario, I want to use the library to update the CUE points within a node.js local processed script. The files I am editing are podcasts that are 1 hour long - thus the WAV files tend to be around 500MB. The library seems to only handle smaller files (I am getting the
JavaScript heap out of memorywith the big file, but not with a 1-minute file that's only 15MB in size).So my question is two-fold:
I'd really love to be able to use this lib. It's an amazing project and it would be a lifesaver for me. Sadly right now I can't really make much use of it.
Looking forward to working with WaveFile!
Chris
P.S.: I also tried setting the
node --max-old-space-size=8192to ridiculously high values. Nothing worked.