ulixee / secret-agent

The web scraper that's nearly impossible to block - now called @ulixee/hero
https://secretagent.dev
MIT License
675 stars 45 forks source link

TypeError: Cannot read properties of null (reading 'type') #459

Open mquadrant opened 2 years ago

mquadrant commented 2 years ago

I am getting error when running the scrapper function in a cron, and throw the error at the second run.

TypeError: Cannot read properties of null (reading 'type') at Object.createResource

async function secretAgentFunction({ url }) {
    await agent.goto(url);
    console.log('Running ----- ', url)
    await agent.close();
}
blakebyrnes commented 2 years ago

@mquadrant - I can't see your whole code snippet, but are you using the default agent that comes from const agent = require('secret-agent')? If you are, and this function is being called twice, it looks like you might be closing it and then trying to reuse it. You should move this to use a handler, or just construct a new Agent each time. Hopefully this helps!

mquadrant commented 2 years ago

@blakebyrnes I tried using the handler without closing the handlers and it works for 10 runs and then it throws this MaxListenersExceededWarning: Possible EventEmitter memory leak detected. While closing the agent/handler throw TypeError: Cannot read properties of null (reading 'type') at Object.createResource on the second run.

blakebyrnes commented 2 years ago

The max listeners message is just a warning. We probably need to look into that, but it shouldn't be causing any side effects.

Would you might sharing a full code sample and which url you're seeing this on? (You're welcome to send to me on or Discord channel if you're not comfortable putting on github).