search

fergiemcdowall / norch-fetch

Fetch pure HTML from a webserver and save it to disk
MIT License
8 stars 2 forks source link

User agent string #10

Open eklem opened 10 years ago

eklem commented 10 years ago

Not sure what norch-fetch identifies as when crawling a site, but it should be possible to define which type of Agent (norch-fetch), version and who is using it (with a link to a page explaining the purpose).

-u --useragent telling the site you crawls, who is crawling. Default is norch-fetch [version]

eklem commented 10 years ago

And here's an formatting example: http://en.wikipedia.org/wiki/User_agent#Format_for_automated_agents_.28bots.29