I've prepared a branch with some modifications I had to include in the codebase in order to provide custom WebURLs with adittional fields (and the necesary diferent WebURLTupleBindings).
In order to achieve that, I had to introduce the following modifications:
WorkQueues creates a WebURLTupleBinding. I added a protected constructor which accepts it as a parameter, so superclases can provide custom WebURLTupleBinding instances.
Frontier creates the WorkQueues in the constructor. Now it has a createWorkQueues method that can be overriden by subclases in order to create custom subclases.
Same than above for CrawlController and methods createFrontier createEmptyWebURL
WebCrawler class logic to follow redirections and outgoing URLs is now placed inside protected functions that can be overriden.
I've prepared a branch with some modifications I had to include in the codebase in order to provide custom WebURLs with adittional fields (and the necesary diferent WebURLTupleBindings).
In order to achieve that, I had to introduce the following modifications: