Open Basuri opened 5 years ago
Hi,
I build a crawler with this library. But when I comparing with some other online crawlers, this library not listing all urls. Some times this library crawl only 100 urls. How can I maximise this?
Hi,
I build a crawler with this library. But when I comparing with some other online crawlers, this library not listing all urls. Some times this library crawl only 100 urls. How can I maximise this?