markkvdb commented 3 years ago

Allow exact matches and create greater abstraction of the base scraper class

Description

80 requests support for exact search queries. This is now incorporated by a different way of constructing the url to search. An option is added to scraper (--exact-result for CLI and exact_result in the YAML) which activates the exact query. Basically this just means adding quotes around the query.
The search url constructing has been adapted to simplify adding more locals. For non-english locales, the search url is slightly different for Monster. This can now be set with one-line. Furthermore, if a locale has unique search options than this can also be added with one line.
While working on the search url construction I observed that more functionality built into the scrapers could be moved to the base scraper. This comes from the following observation: every scraper starts by collecting the number of jobs and pages (get_n_pages) and then proceeds to obtain all search pages (get_job_soup_page) and listings on those pages (_parse_job_listings_to_bs4).

Context of change

Please add options that are relevant and mark any boxes that apply.

[ ] Software (software that runs on the PC)
[x] Library (library that runs on the PC)
[ ] Tool (tool that assists coding development)
[ ] Other

Type of change

Please mark any boxes that apply.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?

The only real new functionality is allowing for exact queries. Testing this consists of making sure that the CLI flag and YAML entry are properly handled. This was easy since I just follow the procedure of similar_results. The actual change in the code base to allow this was a single line to add quotes to query, hence little testing is required.

Checklist:

Please mark any boxes that have been completed.

[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.
[x] Any dependent changes have been merged and published in downstream modules.

codecov-commenter commented 3 years ago

Codecov Report

Merging #114 into master will increase coverage by 1.80%. The diff coverage is 37.50%.

@@            Coverage Diff             @@
##           master     #114      +/-   ##
==========================================
+ Coverage   35.72%   37.53%   +1.80%     
==========================================
  Files          22       22              
  Lines        1447     1420      -27     
==========================================
+ Hits          517      533      +16     
+ Misses        930      887      -43

Impacted Files	Coverage Δ
jobfunnel/config/manager.py	`30.43% <ø> (+0.64%)`	:arrow_up:
jobfunnel/config/settings.py	`66.66% <ø> (ø)`
jobfunnel/backend/scrapers/base.py	`36.01% <31.66%> (-3.48%)`	:arrow_down:
jobfunnel/backend/scrapers/monster.py	`32.11% <38.09%> (+5.07%)`	:arrow_up:
jobfunnel/backend/scrapers/indeed.py	`34.10% <40.00%> (+7.11%)`	:arrow_up:
jobfunnel/backend/scrapers/glassdoor.py	`36.60% <42.85%> (+5.67%)`	:arrow_up:
jobfunnel/config/cli.py	`89.47% <100.00%> (+0.14%)`	:arrow_up:
jobfunnel/config/search.py	`77.14% <100.00%> (+0.67%)`	:arrow_up:
jobfunnel/resources/defaults.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a31e09f...616d351. Read the comment docs.

PaulMcInnis commented 3 years ago

Closing this for now due to inactivity, 100% open to revisiting this.

PaulMcInnis / JobFunnel