Don't issue HTTP requests without a 'user agent' header

rubys / venus

Planet Venus is an awesome ‘river of news’ feed reader. It downloads news feeds published by web sites and aggregates their content together into a single combined feed, latest news first.

http://intertwingly.net/code/venus/docs/index.html

Other

273 stars 99 forks source link

Don't issue HTTP requests without a 'user agent' header #29

Open jetvillegas opened 10 years ago

jetvillegas commented 10 years ago

WP Engine (and perhaps other blog hosts) reject HTTP requests that don't have a user-agent field in the header. Planet Venus issues such requests for feeds when spider_threads > 0.

jetvillegas commented 10 years ago

Actually, what the server is rejecting is the default user-agent sent by the outdated httplib2 in the venus tree: "user-agent: Python-httplib2/$Rev$"

The latest httplib2 seems to have fixed this issue.