search

amoilanen / js-crawler

Web crawler for Node.JS
MIT License
253 stars 55 forks source link

How to deal with ETIMEDOUT error and pending forever? #47

Open fanzijian opened 7 years ago

fanzijian commented 7 years ago

Hello, Thanks for the nice robust crawler. But I got ETIMEDOUT error sometimes. Besides, some http request may stay in pending state and the request died after 5 minutes.

Thanks fanzijian

tibetty commented 7 years ago

Please refer to my modification over js-crawler from https://github.com/tibetty/ya-js-crawler/blob/master/crawler.js, line#406 & line#450

amoilanen commented 7 years ago

Thanks @tibetty We may need to include the modifications from https://github.com/tibetty/ya-js-crawler/blob/master/crawler.js into the default branch of https://github.com/antivanov/js-crawler/ and then we can publish them to NPM. I will take a look

amoilanen commented 7 years ago

And thanks to @fanzijian for reporting the issue. Will try to get the fix published soon

tibetty commented 7 years ago

@antivanov I had customized it too much to fit into my own purpose. I will try to figure out some useful but not so disruptive features and integrate into this project.

amoilanen commented 7 years ago

Thanks @tibetty, please create a pull request if there is time and some changes to contribute to the default branch of the project