search

kazuar / login_scraper_example

214 stars 137 forks source link

Unable to run this code. Getting empty bucket_list #2

Closed nehalbhanushali closed 7 years ago

nehalbhanushali commented 7 years ago

@kazuar I copied the exact same code. is there something I am missing?

nehalbhanushali commented 7 years ago

Also, while I am able to see the login happening in bitbucket example, I am not able to login in other examples

kazuar commented 7 years ago

@nehalbhanushali, thanks for the feedback.

1) Bitbucket changed the website a little so I had to change the url for the repositories list. You can get the new changes and check again. 2) This type of login will only work on some sites. What other sites did you try?

nehalbhanushali commented 7 years ago

@kazuar, I tried on https://freddiemac.embs.com/FLoan/Data/download.php

kazuar commented 7 years ago

Interesting, there are a couple of options here:

1) Try to use firebug or chrome debug tools and check the requests and response while performing login to the site. It might be that you need to set a specific header / user agent when sending the login credentials to the server. 2) Another option is to use selenium for the login and scraping and it will imitate a user opening a browser and entering his credentials. Are you familiar with Selenium?

kazuar commented 7 years ago

Closing as there's no further communication.