Out of memory error

[doug-wade]

dougwade main/corvair ‹dbw-test-gulp-generate-components*› » jscpd -e target/ -e node_modules/
info:    jscpd - copy/paste detector for programming source code, developed by Andrey Kucherenko
info:    Preprocessors running time: durationMs=12123
info:    Scanning 37213 files for duplicates...

<--- Last few GCs --->

  715581 ms: Mark-sweep 1378.7 (1441.2) -> 1378.7 (1441.2) MB, 3159.8 / 0 ms [allocation failure] [GC in old space requested].
  718731 ms: Mark-sweep 1378.7 (1441.2) -> 1378.7 (1441.2) MB, 3150.1 / 0 ms [last resort gc].
  721964 ms: Mark-sweep 1378.7 (1441.2) -> 1378.7 (1441.2) MB, 3233.3 / 0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2f3b9cac9e59 <JS Object>
    2: generateMap [/usr/local/lib/node_modules/jscpd/lib/tokenizer/TokenizerCodeMirror.js:~84] [pc=0xfd58aa1a903] (this=0x5da95c5ee79 <a TokenizerCodeMirror with map 0x1606a80801>)
    3: detect [/usr/local/lib/node_modules/jscpd/lib/strategy.js:39] [pc=0xfd58a972ec7] (this=0x5da95c5efe1 <a Strategy with map 0xf0a72e716b1>,map=0x5da95c5ef61 <a Map with map 0xf0a72e72159>,file=0x255070004ad9 <S...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
[1]    3952 abort      jscpd -e target/ -e node_modules
dougwade main/corvair ‹dbw-test-gulp-generate-components*› » node -v                                                                                                        134 ↵
v6.2.2

[kucherenko]

thank you for report, will investigate and fix

[agirorn]

I encountered this issues today and solved it by increasing the amount of memory node.js uses.

node --max-old-space-size=8192 node_modules/.bin/jscpd

It might be a nice idea to add it to the README file.

[kucherenko]

Just released jscpd@2.0.0 with persistence db inside, for now, you can run jscpd on really big projects

[nishwalshetty]

facing this issue while running jscpd scan for big repo in jenkins ,Could you just tell me where the changes needs to be made exactly ?

[kucherenko]

@nishwalshetty could you please provide more information about jscpd version and node.js env?

[nishwalshetty]

@kucherenko jscpd version 2.0.16 and it is running in linux environment with node 8.15.0

[himanshu2416]

<--- Last few GCs --->

[17914:0x307ba90] 3918376 ms: Mark-sweep 1369.0 (1463.0) -> 1368.6 (1471.0) MB, 350.7 / 0.1 ms allocation failure GC in old space requested [17914:0x307ba90] 3918678 ms: Mark-sweep 1368.6 (1471.0) -> 1368.6 (1433.0) MB, 301.4 / 0.4 ms last resort GC in old space requested [17914:0x307ba90] 3918992 ms: Mark-sweep 1368.6 (1433.0) -> 1368.6 (1432.0) MB, 313.7 / 0.1 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3258c24258b9 0: builtin exit frame: entries(this=0x3258c2404831 <JSFunction Object (sfi = 0x3bf429509cf9)>,0x5b4b37d9f01

[himanshu2416]

Hi @kucherenko , above is the error stack we are facing when we(@nishwalshetty) did the scan on our projects.

[kucherenko]

Hi @himanshu2416, thank you for the report, I will investigate, looks like I should reopen the issue.

[nishwalshetty]

@kucherenko did you find any workaround

Hi @kucherenko , above is the error stack we are facing when we(@nishwalshetty) did the scan on our projects.

@kucherenko did you find any workaround for this ?

[kucherenko]

try to upgrade version of node.js to latest

[kucherenko]

also you can try to use command:

node --max-old-space-size=8192 node_modules/.bin/jscpd

[nishwalshetty]

@kucherenko no luck with latest node version 10.17.0 also the above command was also tried. please find the stacktrace below

jscpd --min-tokens 200 --reporters html --output . --mode strict --format javascript,java --ignoreCase true . node --max-old-space-size=8192 node_modules/.bin/jscpd

<--- Last few GCs --->

[5438:0x39c01e0] 12884970 ms: Mark-sweep 1393.3 (1445.2) -> 1393.3 (1414.2) MB, 138.9 / 0.0 ms (average mu = 0.143, current mu = 0.000) last resort GC in old space requested [5438:0x39c01e0] 12885129 ms: Mark-sweep 1393.3 (1414.2) -> 1393.3 (1414.2) MB, 159.3 / 0.0 ms (average mu = 0.080, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x187f5c87e441]

Security context: 0x2f870819e6e9 1: _put [0xcd0240ef389] [/usr/lib/node_modules/jscpd/node_modules/leveldown/leveldown.js:~52] [pc=0x187f5f528491](this=0x031ff1c3bdf1 ,key=0x34ad72916221 <String[24]: 2019-11-13T11:30:50.721Z>,value=0x33582d33c809 <Very long string[9838981]>,options=0x33582d33ca49

[kucherenko]

jscpd --min-tokens 200 --reporters html --output . --mode strict --format javascript,java --ignoreCase true . node --max-old-space-size=8192 node_modules/.bin/jscpd

Is not right command, the following is right:

 node --max-old-space-size=8192 node_modules/.bin/jscpd --min-tokens 200 --reporters html --output ./report/ --mode strict --format javascript,java --ignoreCase .

[kucherenko]

fixed at jscpd@3.3.0-rc.9

[XiongbinZhao]

@kucherenko Hi, I just ran into this issue with jscpd: 3.3.0-rc.10, node: v14.5 And my Mac has 64 GB memory and I tried node --max-old-space-size=12288 node_modules/.bin/jscpd, but didn't help.

I googled around and found this on StackOverflow, saying this might happen because of one single oversized object, like Map or Array https://stackoverflow.com/questions/55465821/getting-heap-out-of-memory-error-even-when-available-heap-memory-is-much-larger

It would be greatly appreciated if you could reopen this issue and look into it. Thanks for the great work you put into maintaining this project! !

Cheers

<--- Last few GCs --->

[80419:0x108008000] 303959 ms: Mark-sweep 4069.1 (4121.4) -> 4053.5 (4121.9) MB, 6384.0 / 0.0 ms (average mu = 0.102, current mu = 0.001) allocation failure scavenge might not succeed [80419:0x108008000] 311113 ms: Mark-sweep 4069.7 (4121.9) -> 4054.8 (4123.6) MB, 7077.7 / 1.5 ms (average mu = 0.053, current mu = 0.011) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0x100bfea43 node::Abort() (.cold.1) [/usr/local/bin/node] 2: 0x10008609d node::FatalError(char const, char const) [/usr/local/bin/node] 3: 0x100086206 node::OnFatalError(char const, char const) [/usr/local/bin/node] 4: 0x10018cc95 v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [/usr/local/bin/node] 5: 0x10018cc3f v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [/usr/local/bin/node] 6: 0x1002b2ff1 v8::internal::Heap::FatalProcessOutOfMemory(char const) [/usr/local/bin/node] 7: 0x1002b4354 v8::internal::Heap::MarkCompactPrologue() [/usr/local/bin/node] 8: 0x1002b1d3b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node] 9: 0x1002b04e4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node] 10: 0x1002b8654 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node] 11: 0x1002b86aa v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node] 12: 0x1002985a3 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node] 13: 0x10050a25f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long, v8::internal::Isolate*) [/usr/local/bin/node] 14: 0x10077d659 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node] 15: 0x1007ff6ab Builtins_CreateObjectLiteralHandler [/usr/local/bin/node] [1] 80419 abort jscpd --min-lines 40 -r json,html -o “/Users/jenkins_node/Sites” -b

[kucherenko]

I suggest install @jscpd/leveldb-store

npm i @jscpd/leveldb-store

and run following command

jscpd --store leveldb /path/to/files

[XiongbinZhao]

I suggest install @jscpd/leveldb-store

npm i @jscpd/leveldb-store

and run following command

jscpd --store leveldb /path/to/files

@kucherenko Hi, thanks for the quick response!

I see a warning store name leveldb not installed. and end up with the memory issue again, but I already ran npm install @jscpd/leveldb-store --save and also I can see leveldb is intalled when I run npm list. Did I miss something?

└─┬ @jscpd/leveldb-store@3.3.0-rc.10
  ├─┬ @jscpd/core@3.3.0-rc.10
  │ └── eventemitter3@4.0.4
  ├─┬ @jscpd/tokenizer@3.3.0-rc.10
  │ ├── @jscpd/core@3.3.0-rc.10 deduped
  │ ├── reprism@0.0.11
  │ └── spark-md5@3.0.1
  ├─┬ fs-extra@9.0.1
  │ ├── at-least-node@1.0.0
  │ ├── graceful-fs@4.2.4
  │ ├─┬ jsonfile@6.0.1
  │ │ ├── graceful-fs@4.2.4 deduped
  │ │ └── universalify@1.0.0 deduped
  │ └── universalify@1.0.0

[kucherenko]

Could you please tell me the command what you run for detect duplications? Is it globally installed jscpd or locally?

[XiongbinZhao]

Thanks for pointing this out! I just figured out I installed leveldb locally while jscpd was installed globally, after I install them both globally, it ran successfully!

Thanks again for the help! Appreciate