Custom filtering

[suntong]

Hi,

I like your "structured scraper" concept very much. It really captures what scraping is actually doing.

Please keep up the good work, especially, make goscrape more extensible, allowing people to customize nearly every step of processes, including:

• a mechanism of deduplication. This is my original thought. Defining a function that users can override is one of only few things I can think of from my limited go architectural knowledge. Moreover,
• a mechanism of "Custom filtering". E.g., to only scrap those that contain certain keywords, and/or, to discard those that contain certain keywords. This is along the same line or deduplication, but broader.

Hope you can come up something that covers above. I think the feature will be very useful.

Thanks

[suntong]

Please refer to https://github.com/suntong/goscrape/issues/6 for the fix.