search

PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.
MIT License
1.78k stars 210 forks source link

Add driving distance field #131

Closed Stedag closed 3 years ago

Stedag commented 3 years ago

Is your feature request related to a problem? Please describe. I'd like to sort jobs by distance so I set up a script to add the distance of a job to my home.

Describe the solution you'd like I'd like to add optional configuration keys, google api key and home location. If those parameters are available, jobfunnel adds the driving distance value (in minutes) to an extra field in the csv.

Describe alternatives you've considered I've written a script to add the distance field to the resultant pandas dataframe.

Additional context na

thebigG commented 3 years ago

Sorry, I might have to play devil's advocate here.

We already have something similar to what you want to do: radius.

I can see this being a useful feature, but I have a few concerns.

  1. You'll never get the exact address of a job. This is because job sites such as indeed and monster never give you the exact location, they usually just give you the city/town of the job. So your best bet is to get the time it takes you to get from city A to city B. And that can be wildly inaccurate if either city is big. You could get clever and get the exact address through the Google API of the job, but this will add complexity to JobFunnel that might be very painful to write unit tests for.

  2. The setup you propose(especially the Google API key) might not be very user-friendly. This means users that want to use this feature will need to go on some Google website, sign up/sign in, get their API key. We could document all this, but it can get painful to maintain this documentation as we will need to keep up with any changes in the future of this process of obtaining an API key.

  3. Using a Google API key for testing might be problematic if we want to integrate said tests into our CI pipeline. What if someone deactivates that key? What if the service is down at that moment in time, etc.

  4. Privacy. I have to say it as a user of JobFunnel. I understand that users would be able to opt out of using Google API. However, I fear this might make some users skeptical (including myself) about using the tool.

Now if the community wants this really bad, then cool. But just wanted to point out my concerns with implementing this functionality. So I think we need to really hear from the community before considering doing this.

Stedag commented 3 years ago

You're right. Radius is enough.

That being said, I found that I'm unlikely to get permission to scrape so I won't bother. I know it's a matter of toothless policy but I'll wait till they come up with more reasonable terms. :/