search

Bunsly / JobSpy

Jobs scraper library for LinkedIn, Indeed, Glassdoor & ZipRecruiter
https://usejobspy.com
MIT License
576 stars 114 forks source link

job descriptions truncated #11

Closed katchia closed 5 months ago

katchia commented 10 months ago

Running into an issue where job descriptions are truncated when pulling them. Any idea why?

minicoz commented 10 months ago

As I searched on zip recruiter htmls structure, it seems like its broken up into paragraphs and the code pulls in the first paragraph it finds.

cullenwatson commented 10 months ago

The full job description requires another GET request as it's only visible from what I can see at each individual job posting url. Got it working for ZipRecruiter, going to add the others first before I merge.

cullenwatson commented 10 months ago

Issue resolved in PR #14

atseng17 commented 5 months ago

Seems to be happening again, when pulling up 20 results, I saw some are truncated and some are not. It seems only a part of the paragraph is pulled(not the first paragraph).

cullenwatson commented 5 months ago

@atseng17 yes, it was removed due to rate limiting as you have to send a request to each job posting to get the full description, significantly slowing the script down. Probably a better alternative is a flag to fetch the full description so that the people that just want the job titles/urls can quickly fetch them. What do you think?

atseng17 commented 5 months ago

Got it! yeah that makes sense! Thank you for the immediate reply! Having the flexibility is nice.

I imagine there will be ppl who want to get the full description, and for those ppl, they probably are okay with the slow down in fetching descriptions, while others who just need the titles, then don't really need the descriptions.