spinlud / py-linkedin-jobs-scraper

MIT License
307 stars 84 forks source link

li_at token expires after 15 minutes #76

Open kalmanlukelin opened 6 months ago

kalmanlukelin commented 6 months ago

Hi, it seems that li_at token would expire after 15 minutes. Although it says that it would expire in a year in the official doc. Any suggestion to extend the lifetime of the cookie instead of refreshing it manually? https://www.linkedin.com/legal/l/cookie-table

spinlud commented 5 months ago

Hi, maybe you could listen to the invalid session event https://github.com/spinlud/py-linkedin-jobs-scraper/blob/master/linkedin_jobs_scraper/events/events.py#L11 and try a programmatic login (e.g. using Selenium) to Linkedin to obtain a new session cookie.

calvinomiguel commented 5 months ago

On my end, it's expiring even sooner. No matter how slow I set the slow_mo value too. I started with 5 now I am at 100. By the way, I am authenticated

phillips-systems commented 1 month ago

This may be related to the random UA string being used. After replacing the list of UA strings with the exact one from my browser my session remains valid ~2 hours in.

spinlud commented 1 month ago

I have removed the random user agent override as suggested by @phillips-systems, let me know if this helps a bit. Unfortunately it seems that Linkedin expires earlier sessions that appear 'suspect' (according to some criteria that are unclear). Using the same browser to obtain the cookie and run the scraper could probably help...

The same issue is mentioned here https://www.reddit.com/r/webscraping/comments/18t2xao/issues_with_linkedin_li_at_cookie_expiration/

phillips-systems commented 1 month ago

If I had to guess, they are validating cookie values against UA strings as a method to prevent account takeover.

rshah2399 commented 1 month ago

On my end, it's expiring even sooner. No matter how slow I set the slow_mo value too. I started with 5 now I am at 100. By the way, I am authenticated

how did you get authenticated, I cant seem to get there either