Consider using CrawlerTasks instead of raw URL objects for tasks. This will
give the crawler the ability to:
1)Download tasks that are post requests to a web forms with specific post
params.
2)Send requests with custom headers on some websites
and the last and most important:
3)If the Parser wants to add custom information about the url at the moment,
there is no way to do it, because it sets standard java.net.URL on the Page
object.
Here is a patch that adds supports for tasks to the Page object. It also keeps
the information about the seed every task came from.
Original issue reported on code.google.com by nikolavp@gmail.com on 22 Jun 2011 at 10:58
Original issue reported on code.google.com by
nikolavp@gmail.com
on 22 Jun 2011 at 10:58Attachments: