Monster results contain CSS in blurb field

PaulMcInnis commented 4 years ago

Description

Hey everyone, I was gonna cut us a new release, but I noticed an issue:

Currently you get blurbs like below for all jobs scraped from Monster with GlassdoorStatic scraper:

.css-1noe2rc *{color:#505863;line-height:1.4em;}.css-1noe2rc .ecgq1xb1{padding-left:0;}.css-1noe2rc .ecgq1xb1 .ecgq1xb0{margin:0 0 8px 0;}.css-1noe2rc ol,.css-1noe2rc ul{padding-left:32px;}.css-1noe2rc li{margin:10px;margin-bottom:5px;margin-left:20px;line-height:1.4em;}.css-58vpdc{margin-bottom:24px;}.css-58vpdc ul{margin:5px 0 10px 20px;}.css-58vpdc ul > br{display:none;}.css-58vpdc ul > li{margin-left:0;}.css-58vpdc li{padding:0;}PlayStation isn't just the Best Place to Play it's also the Best Place to Work. We've thrilled gamers since 1994, when we launched the original ...

Since glassdoor runs first most of the duplicates are in other job sites and as a result most of the jobs I scrape now have blurbs like the one above.

I was just checking out GlassDoorDynamic and it seems to work well but it misses the date and blurb fields for jobs. As a side note, watching the browser windows go by made me feel like I was in the matrix 😎

Perhaps it is easier for us to purge some of the CSS from these blurbs in the GlassDoorStatic scraper with a regex for longest string in the raw scrape? Open to suggestions.

Alternatively we could just:

rename blurb to something more 'machiney' to reflect that it's not gonna be human readable all the time
switch to scraping indeed first in default settings so most of the burbs will contain human readable blurbs.

Steps to Reproduce

Easily replicable on the stock YAMl on current master with command funnel -kw Engineer, resulting ./search/masterlist.csv will contain aforementioned results.

Expected behavior

blurb should not contain CSS

Actual behavior

blurb contains CSS.

Environment

Build: current Master b30b28453a0f3528095166d0cbbe871726929b64
Operating system and version: OSX, Python3 w/ fresh install in a Virtualenv.
[Linux] Desktop Environment and/or Window Manager: n/a

bunsenmurder commented 4 years ago

Hey @PaulMcInnis are you still getting this issue? I was unable to reproduce it.

PaulMcInnis commented 4 years ago

I just checked now on current master with default settings.yaml, was unable to repro - I'll close this issue and cut a new release.

PaulMcInnis / JobFunnel