PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.
MIT License
1.83k stars 214 forks source link

Improved remote job search support #105

Open LilySu opened 4 years ago

LilySu commented 4 years ago

Description

It would be great to not search by city, as even if the filter can be set for remote, the results returned are still based on city indicated.

Steps to Reproduce

  1. Include '-Indeed' in 'providers:' in the my_settings.yaml
  2. run funnel load -s my_settings.yaml

Expected behavior

Successfully scrape and generate .csv

Actual behavior

Traceback and ValueError if 'province_or_state' or 'city' is not included

Environment

PaulMcInnis commented 4 years ago

Agreed, I think we can do away with city and perhaps also province in the url entirely if its a Fully remote search.

PaulMcInnis commented 4 years ago

per discussion in #106, lets take the approach of allowing city and province/state to be None, with a default of '' in our CSV entries when we write out jobs.

This is valid not only for explicitly remote searches, but also for users who are interested in locating jobs anywhere in a country.

I think it may also make sense to allow for globally-remote searching, but we currently dont' have the appropriate search queries for our providers (which are localised to country). I think it makes sense to leave that feature for when we implement a provider for globally-remote jobs.

apontejosea commented 4 years ago

For now, probably we can just add the city and make the search radius ridiculously large.

PaulMcInnis commented 4 years ago

@josian good point about search radius. We may also be able to exclude it from the query URL entirely. Need to look into this.