inspirezonetech / JobSearchWebScraping

Use web scraping with python selenium to extract job postings from website.
https://inspirezone.tech/learn-web-scraping-with-python-in-minutes/
MIT License
2 stars 9 forks source link

feat: send jobs per e-mail #30

Closed AzizCode92 closed 3 years ago

AzizCode92 commented 3 years ago

Please fill in this pull request template before submitting

1. This pull request resolves #29

2. Description

Add configuration file. Send jobs description via gmail.

3. Fill in checklist by marking [x]

funbeedev commented 3 years ago

@Azizcode92 Thanks for your PR. We'll review this soon!

funbeedev commented 3 years ago

@AzizCode92 I've checked your code locally but it doesn't appear to be working for me. When I configure my gmail details and run the script I get the following error:

  File "job-search-web-scraping.py", line 64, in <module>
    indeed_job_search(*sys.argv)
  File "job-search-web-scraping.py", line 53, in indeed_job_search
    gmailer.send(sender=cfg["sender_email"],
  File "/home/JobSearchWebScraping/utils/mail.py", line 44, in send
    msg = MIMEMultipart.MIMEMultipart()
AttributeError: type object 'MIMEMultipart' has no attribute 'MIMEMultipart'

Also, could you explain why the utils/ folder is included with the script? Ideally we want a way of sending emails without having to include source code of the module. There are built in python modules called smtplib and email.message which you can use to send emails from any mail client and not just Gmail.

You can import the functions you need from these modules using:

import smtplib
from email.message import EmailMessage

Can you modify your PR to use these modules instead? It will allow us to send emails from any mail client. I also have a post on my blog explaining how to send emails using these modules if you want to use that as a guideline: https://inspirezone.tech/sending-emails-with-python/

If you have any questions let me know!

AzizCode92 commented 3 years ago

Hi @funbeedev, I will look into your response asap. thanks.

AzizCode92 commented 3 years ago

I got an error when trying to execute the code locally, selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[name="q"]"} Unfortunately, I wasn't able to verify wether the code works locally or not, if it works fine for you let me know.

thanks.

funbeedev commented 3 years ago

@AzizCode92 Indeed.com changed the layout of their site a bit which is what is causing the error you see when you run the script locally. The problem is the URL https://www.indeed.com/worldwide no longer has a search field. Change this to https://www.indeed.com and the enter a city in your country for the search to work.

After I made these changes the job search will work but the email functionality is not working. Could you change the URL as mentioned above and add a relevant city so you can test your changes locally and make sure it works. Thanks. This is the error I'm getting now:

Traceback (most recent call last):
  File "/usr/lib/python3.8/email/message.py", line 208, in attach
    self._payload.append(payload)
AttributeError: 'str' object has no attribute 'append'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "job-search-web-scraping.py", line 106, in <module>
    indeed_job_search(*sys.argv)
  File "job-search-web-scraping.py", line 97, in indeed_job_search
    send_email(cfg["sender_email"], cfg["sender_password"],
  File "job-search-web-scraping.py", line 38, in send_email
    message.attach(part)
  File "/usr/lib/python3.8/email/message.py", line 210, in attach
    raise TypeError("Attach is not valid on a message with a"
TypeError: Attach is not valid on a message with a non-multipart payload
AzizCode92 commented 3 years ago

Hi @funbeedev, I tested the code and it worked for me locally. try it and let me know if everything is ok for you as well. Bests, Aziz

funbeedev commented 3 years ago

@AzizCode92 I checked the script and it works now! Just a few changes to be made as described in my code review comments. Thanks.

funbeedev commented 3 years ago

@AzizCode92 Thanks for making the changes. I made a slight edit to the config file to make all the options blank. All looks good now so I will merge. Thanks for your contribution!