phantombuster / nickjs

Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)
https://nickjs.org
ISC License
501 stars 48 forks source link

Error: "Chrome subprocess killed by signal SIGTERM" in NodeJS while NickJs is executing #19

Closed ghost closed 6 years ago

ghost commented 6 years ago

Hello guys!

You've been doing a great job with NickJS library!

But working with NickJs on NodeJs I had the following server error: "Chrome subprocess killed by signal SIGTERM". In fact, Chrome is killed by SIGTERM kernel process while NickJs tabs are evaluating.

It seems the problem occurs when code open/close tabs for long time, in particular when pages have a lot of content and scripts. Test script is written to reflect the problem on the real project, where pages also run simultaneously in different tabs for long time. During the work CPU and Memory consumption in Task Manager looks fine, so killing the Chrome process is very unexpected.

My environment: NodeJS v8.8.1, NickJs + Desktop Chrome 63.0.3239.108 (Official Build) (64-bit). Additional packages: Bluebird - for advanced Promise usage. OS: Windows 8 (But the same error occurs on Ubuntu 16.04)

Here is my test file 'pureNickTest.js':

const Promise = require('bluebird');
const nick = new NickJs();
const testUrl = "http://edition.cnn.com/2017/12/15/politics/next-alabama-states/index.html";

const runTab = async (url) => {
    const tab = await nick.newTab();
    await tab.open(url);
    await tab.untilVisible("body");
    await tab.inject(`${__dirname}/libs/jquery-3.2.1.min.js`);
    await tab.wait(3000);
    await tab.scrollToBottom();

    const results = await tab.evaluate((arg, callback)=>{
        const blocks = $("body").find('div');
        callback(null, blocks.length);
    });

    await tab.close();
};

const runTestTabs = () => {
    const fakeArray = new Array(100);

    return Promise.each(fakeArray, () => {
        return runTab(testUrl);
    });
}

(async ()=> {
    await runTestTabs();
    console.log("Test complete!");
    nick.exit();
})();

Equivalent code for puppeteer works fine and doesn't crush with this error.

I would appreciate any help on this issue!

paps commented 6 years ago

First of all thank you for the love & detailed issue. And sorry for the delay, these weeks have been busy at Phantombuster!

I can reproduce your issue, for me it receives SIGTERM around tab 70. I don't understand why, like you say, RAM and CPU usage is normal...

I'm looking into it

ghost commented 6 years ago

That's okay! Thanks for your patient look into this issue!

paps commented 6 years ago

Fixed by e339559408e201eca0d18a672092149725caa3dc

kpennell commented 6 years ago

@paps Thx for fixing this. Updated but am now getting Error: timeout: load event did not fire after 10002ms

I'll have to dig into it. Perhaps I'm using old nick API. My code was for .3.0

paps commented 6 years ago

See my answer to your issue #34 :)