missing results compared to the website search

[luitoz]

I want to extract the results from this url : https://www.linkedin.com/jobs/search?f_TPR=r2592000&keywords=java&location=Jefferson%20County%2C%20Alabama%2C%20United%20States&f_WT=2&f_E=4&sortBy=R&geoId=105185483 the website shows 7K results but when I run the script, it only extracts few results(30): `python3 scrape.py

Scraping...: 0%| | 0/30 [00:00<?, ?it/s]`

[daneski13]

Interesting, on my end it is displaying 31 jobs for that link. This scraper assumes no user is currently logged into LinkedIn so be sure to logout. It's not generally a good idea to scrape from a user account as that is against TOS and may lead to a banned or suspended LinkedIn account.

[luitoz]

Thanks for responding quickly! have you considered implementing an authenticated session functionality to extract all the results shown by the website? to avoid getting banned from LinkedIn, users would use an alternative account only for scrape purposes. I tried to use the LI_AT_COOKIE environment variable to simulate an authenticated session, but it didn't work

[haghani2]

Recently noticed that if you are scraping with specific url and you are not logged in, when going through every single listing some of them don't show any expanded description of the listing and I can only see the block listings on the left and most of the expanded listings on the right is missing. Is it just me or it happens to anyone else? Thanks

[daneski13]

Using a separate account for scraping (and using a user account to scrape in general) is heavily against the TOS and is not something I will support.

some of them don't show any expanded description of the listing and I can only see the block listings on the left and most of the expanded listings on the right is missing.

This is a bug on LinkedIn's end, some sort-of problem with their client-side rendering, potentially even intentional to discourage scraping.