search

mike820324 / ComicSearchEngine

A simple node implementation that help search the comic you want.
MIT License
0 stars 0 forks source link

crawler error handling #1

Closed mike820324 closed 9 years ago

mike820324 commented 9 years ago

Sometimes some of the website will not respond, or with a 404, when this kind of situation take place, the crawler just stop, which is not the behavior that I expected. Maybe I should add a pause method inside crawler, which will pause the crawler for a while and then start the request again.

mike820324 commented 9 years ago

also we must handle ECONNRESET Error

mike820324 commented 9 years ago

longjohn for stack trace. http://stackoverflow.com/questions/10814481/how-to-debug-a-socket-hang-up-error-in-nodejs/11542134#11542134