No running instance or connect refused for few minutes

fabienvauchelles / scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers 🕸️. It also smartly handles traffic routing 🔀 to minimize bans and increase success rates 🚀.

http://scrapoxy.io

MIT License

2.04k stars 235 forks source link

No running instance or connect refused for few minutes #9

Closed cp2587 closed 8 years ago

cp2587 commented 8 years ago

Hi,

Fantastic work on this library. We are using it for quite some times now and it works beautifully 99% of the time. However, we still have some weird errors sometimes and for some request:

407 response with 'No running instance found' in the body
500 response with 'Error in proxy request: Error: connect ECONNREFUSED' in the body

My guess is that scrapoxy returns theses errors when it is refreshing its internal map of ip (once every hour, is that correct ?). Do you know how to handle these ? Do we sleep and retry in a while ?

fabienvauchelles commented 8 years ago

Hi Charles,

Thanks for using it.

Indeed, I reproduced this kind of error. It appends after 1 or 2 hour of webscraping. I'm looking at the error.

Fabien.

fabienvauchelles commented 8 years ago

Hi Charles,

Another user reported me the error. After 1 hour, Scrapoxy sends a HTTP 407 error to every requests. However, many instances are alived!

After 2 or 3 minutes, the error disappears.

Is it similar to your error ?

Thanks, Fabien.

cp2587 commented 8 years ago

I never witnessed the error first hand but the logs seems to indicated that we have the same kind of error: The scrapoxy server responds with 407 for about 2-3 minutes and then it goes away.

cp2587 commented 8 years ago

Hello, Any news on this ?

fabienvauchelles commented 8 years ago

Hi Charles,

I didn't exactly reproduce this bug.

However, I corrected 2 others majors bugs:

1 in proxy.js (scrapoxy/tools/install/proxy.js). The instance doesn't crash when they are errors with the proxy connection. => you could repackage your template instance with this new version.
1 in master.js. Scrapoxy doesn't crash when HTTP headers are malformed. => you could upgrade Scrapoxy.

I didn't poke you because I wasn't sure the bug is corrected :)

Could you upgrade Scrapoxy and your instance template ?

The bug doesn't appear anymore on my Scrapoxy cluster.

Thanks ! Fabien.

cp2587 commented 8 years ago

Hello,

This seems to be resolved (i did not see any error of this type for the past several weeks since i updated the library).

fabienvauchelles commented 8 years ago

Good !

I'm closing the issue.