REMitchell / python-scraping

Code samples from the book Web Scraping with Python http://shop.oreilly.com/product/0636920034391.do
4.42k stars 2.48k forks source link

Can't login to Yahoo! for scraping #72

Open haz-q opened 5 years ago

haz-q commented 5 years ago

I'm going through the 2nd ed. of the book now and it's great. I've spent hours upon hours trying to log into Yahoo! with a POST request but I'm being thwarted. First, the program throws a TooManyRedirects error. When I add the keyword arg of allow_redirects=False, apparently I am being redirected anyway to a site with no content:

Output of response_obj.text: '<p>Found. Redirecting to <a href="https://guce.yahoo.com/consent?gcrumb=F12tPO4&amp;trapType=login&amp;done=https%3A%2F%2Fwww.yahoo.com%2F&amp;src=">https://guce.yahoo.com/consent?gcrumb=F12tPO4&amp;trapType=login&amp;done=https%3A%2F%2Fwww.yahoo.com%2F&amp;src=</a></p>'

I am passing my browser headers and just about every other data I can identify under normal login circumstances with the request. If anyone can successfully log into Yahoo, please spread the knowledge!