Proxies are now supported for scraping the job websites.
The most important change is that all JobFunnel objects do now have a session member s. Therefore, all scrapings for a provider is done within one requests.Session.
Furthermore, I added an optional proxy configuration to the settings.yaml. Unlike all other settings, the proxy settings can be commented out.
Context of change
Please add options that are relevant and mark any boxes that apply.
[x] Software (software that runs on the PC)
[ ] Library (library that runs on the PC)
[ ] Tool (tool that assists coding development)
[ ] Other
Type of change
Please mark any boxes that apply.
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update
How Has This Been Tested?
Running the jobfunnel without a proxy works as expected.
Checklist:
Please mark any boxes that have been completed.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[ ] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[ ] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.
[ ] Any dependent changes have been merged and published in downstream modules.
Resolved the minor issues. I also added unit tests for the url tools. Now that I'm thinking about it I think it's a good idea to check the config dictionary after the argparse. I will create an issue for this.
Pull Request Template
Description
Proxies are now supported for scraping the job websites.
The most important change is that all
JobFunnel
objects do now have a session members
. Therefore, all scrapings for a provider is done within one requests.Session.Furthermore, I added an optional proxy configuration to the settings.yaml. Unlike all other settings, the proxy settings can be commented out.
Context of change
Please add options that are relevant and mark any boxes that apply.
Type of change
Please mark any boxes that apply.
How Has This Been Tested?
Running the jobfunnel without a proxy works as expected.
Checklist:
Please mark any boxes that have been completed.