PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.
MIT License
1.78k stars 210 forks source link

Wage scraping + ensure that we always return in get() + saver URL handling #104

Closed PaulMcInnis closed 3 years ago

PaulMcInnis commented 3 years ago

Description

This adds wage scraping to Indeed and Monster. This data is currently not very reliable, so it provides no filtering.

Context of change

Type of change

How Has This Been Tested?

Self-tested + added this to travis CI

Checklist:

codecov-commenter commented 3 years ago

Codecov Report

Merging #104 into master will decrease coverage by 0.44%. The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #104      +/-   ##
==========================================
- Coverage   36.12%   35.68%   -0.45%     
==========================================
  Files          22       22              
  Lines        1373     1390      +17     
==========================================
  Hits          496      496              
- Misses        877      894      +17     
Impacted Files Coverage Δ
jobfunnel/backend/jobfunnel.py 0.00% <0.00%> (ø)
jobfunnel/backend/scrapers/base.py 37.58% <0.00%> (-0.26%) :arrow_down:
jobfunnel/backend/scrapers/glassdoor.py 29.92% <0.00%> (-0.23%) :arrow_down:
jobfunnel/backend/scrapers/indeed.py 28.57% <0.00%> (-1.35%) :arrow_down:
jobfunnel/backend/scrapers/monster.py 26.38% <0.00%> (-1.15%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ba3a75c...ff66810. Read the comment docs.

PaulMcInnis commented 3 years ago

gonna wait for the CI to pass and I'll merge. Just realized through testing that wage wasn't being read back into CSV again ... oops