search

michalczaplinski / pitchfork

:notes: Unofficial python API for pitchfork.com reviews.
MIT License
73 stars 18 forks source link

Please use a custom User Agent header #1

Closed mattdennewitz closed 10 years ago

mattdennewitz commented 10 years ago

Please use a custom User Agent header so that we know you're playing nice. Thanks!

michalczaplinski commented 10 years ago

Im guessing you mean modifying the default user agent string that is sent by urllib2 "Python-urllib/2.6" to something else? Excuse my ignorance, but what is a purpose of it other than allowing the server admins to see what app reads their page? Is there a commonly accepted practice on what should go in there, like author+app name?

mattdennewitz commented 10 years ago

So that we can see your scraper is exactly why I'm asking. I'll defer to you for what the string could be, but something as simple as michalczaplinski/pitchfork-1.0 would be helpful.

michalczaplinski commented 10 years ago

fixed! thanks for the feedback.