PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.
MIT License
1.85k stars 215 forks source link

Proxy support #51

Closed markkvdb closed 4 years ago

markkvdb commented 4 years ago

Pull Request Template

Description

Proxies are now supported for scraping the job websites.

The most important change is that all JobFunnel objects do now have a session member s. Therefore, all scrapings for a provider is done within one requests.Session.

Furthermore, I added an optional proxy configuration to the settings.yaml. Unlike all other settings, the proxy settings can be commented out.

Context of change

Please add options that are relevant and mark any boxes that apply.

Type of change

Please mark any boxes that apply.

How Has This Been Tested?

Running the jobfunnel without a proxy works as expected.

Checklist:

Please mark any boxes that have been completed.

markkvdb commented 4 years ago

Resolved the minor issues. I also added unit tests for the url tools. Now that I'm thinking about it I think it's a good idea to check the config dictionary after the argparse. I will create an issue for this.