search

fabifont / jobpilot

A straightforward job scraping library
GNU General Public License v3.0
4 stars 1 forks source link

Optimize concurrent fetching for LinkedIn's job endpoints #2

Closed fabifont closed 10 months ago

fabifont commented 11 months ago

LinkedIn has distinct endpoints for retrieving the list of jobs and for fetching specific job details. Each endpoint comes with its own rate limiter.

Given that, it is possible to optimize the data retrieval process by implementing a concurrent fetching mechanism that simultaneously retrieves job listings from one endpoint and individual job details from the other.

This would make full use of the available rate limits for both endpoints, maximizing efficiency and minimizing retrieval time.

fabifont commented 11 months ago

I am currently testing this approach, but it seems that the rate limiter is unified. Probably, the best solution is to fetch all the jobs first and then retrieve all the details.