Can this auto-log in to a website account, and then monitor from there?

nyavramov / monitor_web_page_changes

I thought it would be nice to get an email alert when a new job posting appears on a website, so I can use this to check for the newest job openings on various sites

15 stars 4 forks source link

Can this auto-log in to a website account, and then monitor from there? #1

Open KeronCyst opened 4 years ago

KeronCyst commented 4 years ago

The page I'm trying to track for monitoring is hidden behind a mandatory login.

nyavramov commented 4 years ago

Hey KeronCyst,

I'd need to change the code slightly to allow it to log in to a specific web page. Selenium, the package I'm using in this library, can easily be used to log in to web pages after opening a link.

If you're okay with sharing the website you're trying to log in to, I can update my script allow you to pass your credentials in. If not, I suggest the following:

Open the page you're trying to log in to
Locate the username and password HTML elements using chrome developer tools
Use Selenium to locate and then fill in the text fields of those HTML elements
Use Selenium to press enter or click on login button after you've filled in the credentials

This should work if there's no captcha verification at the login screen. If there is a captcha, things become a whole lot more complicated, though not impossible.

Let me know if you need further help with this.

KeronCyst commented 4 years ago

Thank you so much for your swift response! The website is non-CAPTCHA: https://adminweb.aesoponline.com/access

I was reading https://chrisalbon.com/python/web_scraping/monitor_a_website/ but it doesn't have any login section.

nyavramov commented 4 years ago

No problem. Add this code to line 59 of monitor.py and it should log you in:

time.sleep(3)

username_element = self.driver.find_element_by_id("Username")
password_element = self.driver.find_element_by_id("Password")
login_button_element = self.driver.find_element_by_id("qa-button-login")

username_element.send_keys("YOUR_USERNAME_HERE")
password_element.send_keys("YOUR_PASSWORD_HERE")

login_button_element.click()

KeronCyst commented 4 years ago

Unfortunately, it looks like this particular page seems to go through multiple URLs before finally landing at the one with the actual data that I want to track, and I can't tell if the script is getting there/doing anything after having run it with the addition above, but I appreciate the effort... I should probably mention that I'm a super-newbie to programming languages so figuring this out is probably beyond my league. Thanks anyways, though.