crawler Controller support

mike820324 / ComicSearchEngine

A simple node implementation that help search the comic you want.

MIT License

0 stars 0 forks source link

crawler Controller support #3

Open mike820324 opened 9 years ago

mike820324 commented 9 years ago

Because of issue #1 and issue #2 , I believe it is a much better apporach to add a crawler controller to handle the error and decide whether to stop the crawler or pause the crawler.

The Controller should have the following feature,

send the content to indexer
error handling
start, stop and pause the crawler according to error type

mike820324 commented 9 years ago

Right now, I'm working on a request scheduler so that we don't need to mess up the crawler logic.