ATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[arnaudjnn]

const getProducts = async () => {
    try {
      await Promise.all(
        fs.readdirSync('tmp/awin').map(file => {
          const csvFile = fs.readFileSync(`tmp/awin/${file}`)
          const csvData = csvFile.toString()
          return new Promise(resolve => {
            Papa.parse(csvData, {
              worker: true,
              header: true,
              complete: results => {
                console.log('Complete', results.data.length, 'records.'); 
                resolve(results.data);
              }
            });
          });
        })
      );
    } catch (err) {
      console.error(err);
    }
  }

  getProducts()

return the following error:

<--- Last few GCs --->

[33772:0x104026000]    25042 ms: Mark-sweep 2046.7 (2059.0) -> 2045.9 (2050.7) MB, 57.6 / 0.0 ms  (+ 221.9 ms in 35 steps since start of marking, biggest step 52.0 ms, walltime since start of marking 289 ms) (average mu = 0.151, current mu = 0.055) finali[33772:0x104026000]    25434 ms: Mark-sweep 2047.6 (2050.7) -> 2046.7 (2050.7) MB, 303.3 / 0.0 ms  (+ 81.6 ms in 27 steps since start of marking, biggest step 41.9 ms, walltime since start of marking 392 ms) (average mu = 0.080, current mu = 0.019) alloca

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x100751119]
Security context: 0x2d9b54c80921 <JSObject>
    1: push [0x2d9b54c9a431](this=0x2d9b715517d1 <JSArray[17]>,0x2d9b1b8297b9 <String[194]\: Short par ASOS DESIGN Bient\xf4t dans votre liste d'articles sauvegard\xe9s Motif rayures Passants pour ceinture Fermeture \xe0 boutons Poches c\xf4t\xe9s Deux poches arri\xe8re Coupe classique Taille normalement>)
    2: /* anonymous */ [0x2d9b38dd6911] [/Users/arnaudj...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100bac163 node::Abort() (.cold.1) [/usr/local/Cellar/node/13.12.0/bin/node]
 2: 0x1000802e0 node::FatalError(char const*, char const*) [/usr/local/Cellar/node/13.12.0/bin/node]
 3: 0x100080431 node::OnFatalError(char const*, char const*) [/usr/local/Cellar/node/13.12.0/bin/node]
 4: 0x10017fc41 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/Cellar/node/13.12.0/bin/node]
 5: 0x10017fbeb v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/Cellar/node/13.12.0/bin/node]
 6: 0x1002995af v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/Cellar/node/13.12.0/bin/node]
 7: 0x10029a94c v8::internal::Heap::MarkCompactPrologue() [/usr/local/Cellar/node/13.12.0/bin/node]
 8: 0x100298504 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/Cellar/node/13.12.0/bin/node]
 9: 0x100296fab v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/Cellar/node/13.12.0/bin/node]
10: 0x10029e9f8 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/Cellar/node/13.12.0/bin/node]
11: 0x10029ea4e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/Cellar/node/13.12.0/bin/node]
12: 0x10027c5bf v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/Cellar/node/13.12.0/bin/node]
13: 0x1004e07af v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/Cellar/node/13.12.0/bin/node]
14: 0x100751119 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/Cellar/node/13.12.0/bin/node]
error Command failed with signal "SIGABRT".
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It works if I add to config:

step: function(results) {
   console.log("Product:", results.data.product_name);
},

Why I get this?

[pip77]

I have a pretty similar error, when using parser.pause() and parser.resume() inside the step function. I have set worker: false inside my config.

[jugglinmike]

@arnaudjnn It looks like your input data is too large for your system to hold in memory. When you specify a step function, you enable the tool's streaming mode, so it releases objects as it parses.

@pip77 I saw this too specifically while using a Node.js stream as input. Pausing that stream in addition to the parser solved the problem for me.

 'use strict';
 const fs = require('fs');
 const papaparse = require('papaparse');

 const inFile = fs.createReadStream(process.argv[2], {encoding: 'utf-8'});
 papaparse.parse(inFile, {
   async chunk(row, parser) {
     if (row.errors.length) {
       console.error(row);
       return;
     }
     parser.pause();
+    inFile.pause();
     await new Promise((resolve) => setTimeout(resolve, 2000));
+    inFile.resume();
     parser.resume();
   }
 });

[pokoli]

I'm closing this issue because there is nothing that we need to fix on PapaParse.

Thanks to all for sharing their experiences and solutions.