search

ScottMansfield / widow

Distributed, asynchronous web crawler
GNU Lesser General Public License v2.1
26 stars 4 forks source link

Add ability to restrict crawling to a single domain or domain / sub-domains #12

Closed ScottMansfield closed 9 years ago

ScottMansfield commented 9 years ago

E.g.

aws.amazon.com

or...

*.aws.amazon.com

ScottMansfield commented 9 years ago

addressed in commits

https://github.com/ScottMansfield/widow/commit/d3b94cf569cad9c2af6494ddafd2cc970299c76b https://github.com/ScottMansfield/widow/commit/ed4abeaa3edf0e5e49e05f1a1762a0991faa1633 https://github.com/ScottMansfield/widow/commit/8d4df2684651c7d6fdf5378a041124b1d34a34ec