search

ScottMansfield / widow

Distributed, asynchronous web crawler
GNU Lesser General Public License v2.1
26 stars 4 forks source link

All outbound requests should have a User-Agent attached to them #18

Open ScottMansfield opened 9 years ago

ScottMansfield commented 9 years ago

In Fetch and Parse there are outbound requests using Invocation.blah but have no User-Agent. One in Fetch is fixed but there are others missing.

As well the Terminator and Exo libraries make requests... There should be a way to pass a User-Agent string to them.