search

sujit-kr / crawler4j

Automatically exported from code.google.com/p/crawler4j
0 stars 0 forks source link

missing links in the url list during the crawling process #82

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
when crawling into pages such as this one:
http://steamcommunity.com/games/CSS/members
the links that appears in the browser, at the bottom right corner of the page, 
which allows you to advance into the next pages, are not part of the links in 
the html source.

as you can see in the above link, those pages can supply many links which will 
make crawling very thorough, so it would be a shame to miss them.

can you suggest a way to adjust your source in order to add such functionality?

Original issue reported on code.google.com by yifatcher@gmail.com on 21 Sep 2011 at 11:15