phantombuster / nickjs

Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)
https://nickjs.org
ISC License
500 stars 48 forks source link

Updating to .0.3.6 causes Error: timeout: load event did not fire after 10002ms #34

Closed kpennell closed 6 years ago

kpennell commented 6 years ago

First off, Nick is amazing. Thank you.

I tried upgrading to .0.3.6 (from .0.3.0) get around the Chrome subprocess killed by signal SIGTERM issues.

Now I get a Error: timeout: load event did not fire after 10002ms on my code. Any ideas? The HN example still works so I'm not sure what's wrong with my code.


const Nick = require("nickjs");
const nick = new Nick();

(async () => {
  const tab = await nick.newTab();

  let urls = ["https://www.lawfwfewef.com", "https://www.awfwaffe.com"];

  let outsideresults = [];

  for (let url of urls) {
    const insideresults = [];

    await tab.open(url);

    await tab.wait(9000);

    await tab.untilVisible("html"); // Make sure we have loaded the page
  } // for of loop
})()
  .then(() => {
    console.log("Job done!");
    nick.exit();
  })
  .catch(err => {
    console.log(`Something went wrong: ${err}`);
    nick.exit(1);
  });
kpennell commented 6 years ago

Feel free to close this if you need to. I think it's likely more just related to my code and not the .0.3.6 version.

paps commented 6 years ago

Hey it's not your code. NickJS now handles timeouts correctly for open(). You can change this by setting the timeout option in the constructor. By default it's 10000 which seems to be too low for the sites you're targeting.

kpennell commented 6 years ago

I tried this


new Nick({
  printNavigation: true,
  printResourceErrors: true,
  printPageErrors: true,
  resourceTimeout: 30000,
  userAgent: "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36"
})

But am still getting Something went wrong: Error: timeout: load event did not fire after 10001ms

paps commented 6 years ago

Can you try with timeout instead of resourceTimeout? Sorry... we need to update the readme, which is out of sync with https://hub.phantombuster.com/v1/reference#nick

kpennell commented 6 years ago

Thanks @paps !