We use ABot crawler for crawling an enormous list of websites like 7k ~ 10k a day, and we meet a problem that keep on going since we began to use it.
Sometime the crawler stops and don't continue the code that's behind it, without any error or Exception.
We try to check and catch any http Exception or simple Exception, but it is not enough.
The crawler randomly ( perhaps or we suppose ) stop and the other threads finished their work and the execution stop here and won't continue. It is like we are in an infinite loop or something like that, and we search and didn't find any clues why it stop.
I hope You can help us to find out what is going on.
Hi,
We use ABot crawler for crawling an enormous list of websites like 7k ~ 10k a day, and we meet a problem that keep on going since we began to use it.
Sometime the crawler stops and don't continue the code that's behind it, without any error or Exception. We try to check and catch any http Exception or simple Exception, but it is not enough.
The crawler randomly ( perhaps or we suppose ) stop and the other threads finished their work and the execution stop here and won't continue. It is like we are in an infinite loop or something like that, and we search and didn't find any clues why it stop.
I hope You can help us to find out what is going on.
Thank You.