search

Letractively / flaxcrawler

Automatically exported from code.google.com/p/flaxcrawler
0 stars 0 forks source link

Not Crawling All link #10

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
Hi,

I am using this crawler and extracting required data, but crawler does not 
crawl all the pages for domain. It just crawls around 90-100 web pages and 
automatically gets stop. Even I configured the property, 
configuration.setMaxLevel(0) but still it crawls only limited url.

Can any one help me out how can I crawl all the web pages for 1 domain.

Thank you.

Original issue reported on code.google.com by b.like.no.other on 22 Feb 2012 at 3:57

GoogleCodeExporter commented 8 years ago 
You may need to use "join()" instead of "join(timeout)" which will set up a 
timeout time for the crawling work.

Original comment by tictacto...@gmail.com on 6 Oct 2012 at 6:44