Links with rel="nofollow" should not be followed

ScottMansfield / widow

Distributed, asynchronous web crawler

GNU Lesser General Public License v2.1

26 stars 4 forks source link

Links with rel="nofollow" should not be followed #16

Closed ScottMansfield closed 9 years ago

ScottMansfield commented 9 years ago

Any link with a rel="nofollow" should not be followed, i,e, it should not be parsed or have a HEAD request sent to it even for a content-type evaluation.