The crawler visits only product pages but not the others such as Top Sellers page. Because of that the number of collected links are very limited (It was at most 150 in my trials). The crawler must visit other pages on the site as well as product pages.
rimbi
commented
13 years ago
The crawler visits only product pages but not the others such as Top Sellers page. Because of that the number of collected links are very limited (It was at most 150 in my trials). The crawler must visit other pages on the site as well as product pages.