search

rtfpessoa / diff2html-cli

Pretty diff to html javascript cli (diff2html-cli)
https://diff2html.xyz
MIT License
521 stars 50 forks source link

Still running out of heap memory while allowing node 60GB #140

Closed SichangHe closed 2 years ago

SichangHe commented 2 years ago 
$ NODE_OPTIONS="--max-old-space-size=60000" time diff2html -s side -F ~/diff.html --lm none -- -M HEAD~

<--- Last few GCs --->

[1762257:0x1e0fdf0]   558168 ms: Scavenge 31225.4 (34640.4) -> 31225.3 (34645.9) MB, 72.5 / 0.0 ms  (average mu = 0.951, current mu = 0.951) allocation failure 
[1762257:0x1e0fdf0]   558249 ms: Scavenge 31230.6 (34645.9) -> 31230.6 (34649.9) MB, 74.1 / 0.0 ms  (average mu = 0.951, current mu = 0.951) allocation failure 
[1762257:0x1e0fdf0]   558342 ms: Scavenge 31234.0 (34649.9) -> 31234.0 (34649.9) MB, 87.5 / 0.0 ms  (average mu = 0.951, current mu = 0.951) allocation failure 

<--- JS stacktrace --->
Cannot get stack trace in GC.
FATAL ERROR: Scavenger: semi-space copy Allocation failed - JavaScript heap out of memory
 1: 0x7f36b570c46c node::Abort() [/lib/x86_64-linux-gnu/libnode.so.64]
 2: 0x7f36b570c4b5  [/lib/x86_64-linux-gnu/libnode.so.64]
 3: 0x7f36b5938e6a v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/lib/x86_64-linux-gnu/libnode.so.64]

<--- Last few GCs --->

[1762257:0x1e0fdf0]   558168 ms: Scavenge 31225.4 (34640.4) -> 31225.3 (34645.9) MB, 72.5 / 0.0 ms  (average mu = 0.951, current mu = 0.951) allocation failure 
[1762257:0x1e0fdf0]   558249 ms: Scavenge 31230.6 (34645.9) -> 31230.6 (34649.9) MB, 74.1 / 0.0 ms  (average mu = 0.951, current mu = 0.951) allocation failure 
[1762257:0x1e0fdf0]   558342 ms: Scavenge 31234.0 (34649.9) -> 31234.0 (34649.9) MB, 87.5 / 0.0 ms  (average mu = 0.951, current mu = 0.951) allocation failure 

<--- JS stacktrace --->
Cannot get stack trace in GC.
FATAL ERROR: Scavenger: semi-space copy Allocation failed - JavaScript heap out of memory
 4: 0x7f36b59390e1 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/lib/x86_64-linux-gnu/libnode.so.64]
 1: 0x7f36b570c46c node::Abort() [/lib/x86_64-linux-gnu/libnode.so.64]
 5: 0x7f36b5cd3c66  [/lib/x86_64-linux-gnu/libnode.so.64]
 2: 0x7f36b570c4b5  [/lib/x86_64-linux-gnu/libnode.so.64]
 6: 0x7f36b5d25a0f v8::internal::Scavenger::ScavengeObject(v8::internal::HeapObjectReference**, v8::internal::HeapObject*) [/lib/x86_64-linux-gnu/libnode.so.64]
 3: 0x7f36b5938e6a v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/lib/x86_64-linux-gnu/libnode.so.64]
 7: 0x7f36b5d29277 v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) [/lib/x86_64-linux-gnu/libnode.so.64]
 4: 0x7f36b59390e1 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/lib/x86_64-linux-gnu/libnode.so.64]
 8: 0x7f36b5ccecad v8::internal::ScavengingTask::RunInParallel() [/lib/x86_64-linux-gnu/libnode.so.64]
 5: 0x7f36b5cd3c66  [/lib/x86_64-linux-gnu/libnode.so.64]
 9: 0x7f36b5cf6de3 v8::internal::ItemParallelJob::Task::RunInternal() [/lib/x86_64-linux-gnu/libnode.so.64]
 6: 0x7f36b5d25a0f v8::internal::Scavenger::ScavengeObject(v8::internal::HeapObjectReference**, v8::internal::HeapObject*) [/lib/x86_64-linux-gnu/libnode.so.64]
10: 0x7f36b5cf795a v8::internal::ItemParallelJob::Run(std::shared_ptr<v8::internal::Counters>) [/lib/x86_64-linux-gnu/libnode.so.64]
 7: 0x7f36b5d29277 v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*) [/lib/x86_64-linux-gnu/libnode.so.64]
11: 0x7f36b5cdef0a v8::internal::Heap::Scavenge() [/lib/x86_64-linux-gnu/libnode.so.64]
 8: 0x7f36b5ccecad v8::internal::ScavengingTask::RunInParallel() [/lib/x86_64-linux-gnu/libnode.so.64]
12: 0x7f36b5ce51c7 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/lib/x86_64-linux-gnu/libnode.so.64]
 9: 0x7f36b5cf6de3 v8::internal::ItemParallelJob::Task::RunInternal() [/lib/x86_64-linux-gnu/libnode.so.64]
13: 0x7f36b5ce5930 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/lib/x86_64-linux-gnu/libnode.so.64]
10: 0x7f36b57a2228  [/lib/x86_64-linux-gnu/libnode.so.64]
14: 0x7f36b5ce791d v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/lib/x86_64-linux-gnu/libnode.so.64]
11: 0x7f36b51c4609  [/lib/x86_64-linux-gnu/libpthread.so.0]
15: 0x7f36b5ce7975 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/lib/x86_64-linux-gnu/libnode.so.64]
12: 0x7f36b50e9133 clone [/lib/x86_64-linux-gnu/libc.so.6]
Command terminated by signal 6
756.14user 40.47system 9:22.39elapsed 141%CPU (0avgtext+0avgdata 33122304maxresident)k
8inputs+0outputs (0major+11143007minor)pagefaults 0swaps
SichangHe commented 2 years ago

The diff is HUGE though…

$ git diff HEAD~ > ../diff.diff && .. && ls -lah | grep diff.diff
-rw-rw-r--     1 username username 502M Jul 15 09:35 diff.diff

Am I screwed? 😭

SichangHe commented 2 years ago

Some information about the machine:

$ uname -a
Linux <name> 5.4.0-113-generic #127-Ubuntu SMP <time> x86_64 x86_64 x86_64 GNU/Linux
SichangHe commented 2 years ago

Memory specs:

$ grep MemTotal /proc/meminfo
MemTotal:       98864120 kB
rtfpessoa commented 2 years ago

That diff is HUGE. It is a very weird scenario. I don't have time to dig where might be the problem, if there is even one.