Big Bug - Memory leak when use aso lib

[tracer8]

Hi facundoolano, thank you very much for very useful lib. I use you lib to aso on google play without any problem. But when i try to get score from over 14 keywords sequentially. It came out with error:

<--- Last few GCs --->

692423 ms: Mark-sweep 1256.1 (1434.6) -> 1244.3 (1434.6) MB, 83.0 / 0.0 ms [al location failure] [GC in old space requested]. 692501 ms: Mark-sweep 1244.3 (1434.6) -> 1244.2 (1434.6) MB, 77.8 / 0.0 ms [al location failure] [GC in old space requested]. 692585 ms: Mark-sweep 1244.2 (1434.6) -> 1253.3 (1403.6) MB, 84.1 / 0.0 ms [la st resort gc]. 692663 ms: Mark-sweep 1253.3 (1403.6) -> 1262.2 (1403.6) MB, 78.3 / 0.0 ms [la st resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000000E62C0CFB49 1: slowToString [buffer.js:459] [pc=000002CB180F33F5] (this=0000010137FA80C1 <an Uint8Array with map 000001A00EA067D1>,encoding=000000E62C0DBC31 <String[4]: utf8>,start=0,end=316600) 2: arguments adaptor frame: 1->3 3: toString [buffer.js:~487] [pc=000002CB17FD6C21] (this=0000010137FA80C1 <a n Uint8Array with map 000001A00EA067D1>) 4: arguments adaptor frame: 1->0 5: /* ano...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memo ry`

Then i try to debug this, and find out, when use aso lib, it will take very large memory, and nodejs don't gc this. I try with simple code like this:

var aso = require('aso')('gplay');

// Get memory usage
var heapUsed = process.memoryUsage().heapUsed;
console.log("Program is using " + heapUsed + " bytes of Heap.")

aso.scores("coin").then(function(data)
{
      // Get memory usage
      var heapUsed = process.memoryUsage().heapUsed;
      console.log("Program is using " + heapUsed + " bytes of Heap.")

      aso.scores("fun").then(function(a)
      {
            // Get memory usage
            var heapUsed = process.memoryUsage().heapUsed;
            console.log("Program is using " + heapUsed + " bytes of Heap.")

            aso.scores("game").then(function(b)
            {
                  // Get memory usage
                  var heapUsed = process.memoryUsage().heapUsed;
                  console.log("Program is using " + heapUsed + " bytes of Heap.")

                  aso.scores("coin").then(function(c)
                  {
                        // Get memory usage
                        var heapUsed = process.memoryUsage().heapUsed;
                        console.log("Program is using " + heapUsed + " bytes of Heap.")
                  });
            });
      });
});

Result very bad:

Program is using 80766688 bytes of Heap.
Program is using 213251752 bytes of Heap.
Program is using 390810216 bytes of Heap.
Program is using 756238712 bytes of Heap.
Program is using 796911968 bytes of Heap.

I think many object of aso lib can't destroy when gc working. Hope you will find out what is the issue and fix it. Thank you very much!

[facundoolano]

this is most likely a consequence of an unbounded memoization of the scrapers. We should switch to the most recent version of the scrapers and configure the memoization to limit the amount of cached results.

El ene 18, 2018 11:41 AM, "SpieGame" notifications@github.com escribió:

Hi facundoolano, thank you very much for very useful lib. I use you lib to aso on google play without any problem. But when i try to get score from over 14 keywords sequentially. It came out with error:

`<--- Last few GCs --->

692423 ms: Mark-sweep 1256.1 (1434.6) -> 1244.3 (1434.6) MB, 83.0 / 0.0 ms [al location failure] [GC in old space requested]. 692501 ms: Mark-sweep 1244.3 (1434.6) -> 1244.2 (1434.6) MB, 77.8 / 0.0 ms [al location failure] [GC in old space requested]. 692585 ms: Mark-sweep 1244.2 (1434.6) -> 1253.3 (1403.6) MB, 84.1 / 0.0 ms [la st resort gc]. 692663 ms: Mark-sweep 1253.3 (1403.6) -> 1262.2 (1403.6) MB, 78.3 / 0.0 ms [la st resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000000E62C0CFB49 1: slowToString [buffer.js:459] [pc=000002CB180F33F5] (this=0000010137FA80C1 <an Uint8Array with map 000001A00EA067D1>,encoding=000000E62C0DBC31 <String[4]: utf8>,start=0,end=316600) 2: arguments adaptor frame: 1->3 3: toString [buffer.js:~487] [pc=000002CB17FD6C21] (this=0000010137FA80C1 <a n Uint8Array with map 000001A00EA067D1>) 4: arguments adaptor frame: 1->0 5: /* ano...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memo ry`

Then i try to debug this, and find out, when use aso lib, it will take very large memory, and nodejs don't gc this. I try with simple code like this( add option --expose-gc when execute this):

`var aso = require('aso')('gplay');

// Get memory usage var heapUsed = process.memoryUsage().heapUsed; console.log("Program is using " + heapUsed + " bytes of Heap.")

aso.scores("coin").then(function(data) { // Get memory usage var heapUsed = process.memoryUsage().heapUsed; console.log("Program is using " + heapUsed + " bytes of Heap.")

aso.scores("fun").then(function(a) { // Get memory usage var heapUsed = process.memoryUsage().heapUsed; console.log("Program is using " + heapUsed + " bytes of Heap.")

    aso.scores("game").then(function(b)
    {
          // Get memory usage
          var heapUsed = process.memoryUsage().heapUsed;
          console.log("Program is using " + heapUsed + " bytes of Heap.")

          aso.scores("coin").then(function(c)
          {
                // Get memory usage
                var heapUsed = process.memoryUsage().heapUsed;
                console.log("Program is using " + heapUsed + "

bytes of Heap.") }); }); });

}); `

Result very bad: Program is using 80766688 bytes of Heap. Program is using 213251752 bytes of Heap. Program is using 390810216 bytes of Heap. Program is using 756238712 bytes of Heap. Program is using 796911968 bytes of Heap.

I think many object of aso lib can't destroy when gc working. Hope you will find out what is the issue and fix it. Thank you very much!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/facundoolano/aso/issues/12, or mute the thread https://github.com/notifications/unsubscribe-auth/AA_iLSkaCRRaiFiZ4TAMea_9hNYeFjQpks5tL1g1gaJpZM4RjBj1 .

[tracer8]

So i must install your google-play-scraper with recent update or check your code of aso library to find then configure the memoization to limit the amount of cached results?

Can you show me script name to edit/configure this?

Thanks!

[facundoolano]

The scraper is used by aso here.

And here is the information on how to configure memoization (you can remove it altogether but that will probably slow down some of the aso functions).

[tracer8]

Thank you very much!

[facundoolano]

this should be fixed when #22 is done

[tracer8]

this should be fixed when #22 is done

Awesome! Thank you!