search

josephlimtech / linkedin-profile-scraper-api

🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.
MIT License
566 stars 152 forks source link

What's the best way to update session cookies on a scraper running on server after expiry? #16

Open Aditya94A opened 4 years ago

Aditya94A commented 4 years ago

For me, new free accounts seem to get blocked after 50 profiles/day, sessions seem to randomly expire.

When the session cookie expires, as per the readme, we're supposed to use a new one. But we can't really keep doing this manual step every time when scraping at scale (think millions of accounts), so I'm wondering if there's a way to automate this.

Any ideas?

jvandenaardweg commented 4 years ago

Hmm that seems to be a major limitation then.

Might try LinkedIn's commercial options then? https://www.linkedin.com/help/linkedin/answer/52950

In other words: premium (paid) accounts seem to have higher limits.

For millions of accounts I can imagine you'll have to find other ways to do this. Seems logical LinkedIn tries to prevent that high volume of scraping.