spinlud
commented
3 years ago Hi there! It seems I can't reproduce your issue. I've tried with the following code:
#!/usr/bin/env python
import os
import logging
from linkedin_jobs_scraper import LinkedinScraper
from linkedin_jobs_scraper.events import Events, EventData
from linkedin_jobs_scraper.query import Query, QueryOptions, QueryFilters
from linkedin_jobs_scraper.filters import RelevanceFilters, TimeFilters, TypeFilters, ExperienceLevelFilters, RemoteFilters
def on_data(data):
print(data.job_id)
print(data.title, '|', data.company, '|', data.place, '|', data.date)
print(data.description.replace('\n', ' '))
print(data.link)
print('---------------------------------------------------------------------------------------------------')
def test_run():
# Change other logger levels
logging.basicConfig(
level=logging.INFO,
)
logging.getLogger('li:scraper').setLevel(logging.INFO)
logging.getLogger('requests').setLevel(logging.ERROR)
logging.getLogger('urllib3').setLevel(logging.ERROR)
logging.getLogger('requests.packages.urllib3').setLevel(logging.ERROR)
logging.getLogger('selenium').setLevel(logging.ERROR)
scraper = LinkedinScraper(
headless=False,
max_workers=1,
slow_mo=1.1,
)
scraper.on(Events.DATA, on_data)
scraper.on(Events.ERROR, lambda error: print(Events.ERROR.value, error))
scraper.on(Events.END, lambda: print(Events.END.value))
queries = [
Query(
query='Unity3D',
options=QueryOptions(
# locations=['United States'],
limit=100,
filters=QueryFilters(
# remote=RemoteFilters.REMOTE,
# time=TimeFilters.MONTH
)
)
)
]
scraper.run(
queries=queries,
)
if __name__ == '__main__':
test_run()This is a sample of results running in anonymous mode:
INFO:li:scraper:('Using strategy AnonymousStrategy',)
INFO:li:scraper:('Starting new query', "Query(query=Unity3D options=QueryOptions(limit=100 locations=['Worldwide'] filters=QueryFilters() optimize=False))")
INFO:li:scraper:('[Unity3D][Worldwide]', 'Opening https://www.linkedin.com/jobs/search?keywords=Unity3D&location=Worldwide')
INFO:li:scraper:('[Unity3D][Worldwide]', 'Trying first selectors set')
INFO:li:scraper:('[Unity3D][Worldwide]', 'Trying second selectors set')
INFO:li:scraper:('[Unity3D][Worldwide]', 'OK')
INFO:li:scraper:('[Unity3D][Worldwide]', 'Starting pagination loop')
INFO:li:scraper:('[Unity3D][Worldwide]', 'Found 15 jobs')
2572656639
Game Programmer | Williwaw | Sofia, Bulgaria | 2021-06-04
https://bg.linkedin.com/jobs/view/game-programmer-at-williwaw-2572656639?refId=s9A5LosOF4suvx4eED06%2BA%3D%3D&trackingId=y8FMwi7S2jDfmhlwpH3hSA%3D%3D&position=1&pageNum=0&trk=public_jobs_jserp-result_search-card
---------------------------------------------------------------------------------------------------
INFO:li:scraper:('[Unity3D][Worldwide][1]', 'Processed')
2579030273
Gameplay Programmer | CD PROJEKT RED | Warsaw, Mazowieckie, Poland | 2021-06-02
https://pl.linkedin.com/jobs/view/gameplay-programmer-at-cd-projekt-red-2579030273?refId=s9A5LosOF4suvx4eED06%2BA%3D%3D&trackingId=YCuvgiDLaxV5LItDtOs5lg%3D%3D&position=2&pageNum=0&trk=public_jobs_jserp-result_search-card
---------------------------------------------------------------------------------------------------
INFO:li:scraper:('[Unity3D][Worldwide][2]', 'Processed')
2572679482
Game Developer | GAME PILL | Toronto, Ontario, Canada | 2021-06-04
https://ca.linkedin.com/jobs/view/game-developer-at-game-pill-2572679482?refId=s9A5LosOF4suvx4eED06%2BA%3D%3D&trackingId=TA%2BROy2ra8W1kvkGm1vscw%3D%3D&position=3&pageNum=0&trk=public_jobs_jserp-result_search-card
---------------------------------------------------------------------------------------------------
INFO:li:scraper:('[Unity3D][Worldwide][3]', 'Processed')
2577593534
Game Designer | Xplorro | Chennai, Tamil Nadu, India | 2021-06-05
https://in.linkedin.com/jobs/view/game-designer-at-xplorro-2577593534?refId=s9A5LosOF4suvx4eED06%2BA%3D%3D&trackingId=5eMMa0rkSvmf6JPSfxt0bA%3D%3D&position=4&pageNum=0&trk=public_jobs_jserp-result_search-card
---------------------------------------------------------------------------------------------------
[...]
INFO:li:scraper:('[Unity3D][Worldwide][98]', 'Processed')
2577528416
Intern - Virtual Reality Developer | McKinsey & Company | Wrocław, Dolnośląskie, Poland | 2021-06-04
https://pl.linkedin.com/jobs/view/intern-virtual-reality-developer-at-mckinsey-company-2577528416?refId=MNbgrdvQ2e4l%2BI7u4dN%2B2g%3D%3D&trackingId=wxq1vFhSix6U4b72YQ3uvQ%3D%3D&position=2&pageNum=6&trk=public_jobs_jserp-result_search-card
---------------------------------------------------------------------------------------------------
INFO:li:scraper:('[Unity3D][Worldwide][99]', 'Processed')
2571171580
Software Engineer | Hidden Variable Studios | Los Angeles Metropolitan Area | 2021-05-29
https://www.linkedin.com/jobs/view/software-engineer-at-hidden-variable-studios-2571171580?refId=MNbgrdvQ2e4l%2BI7u4dN%2B2g%3D%3D&trackingId=OoUd4KfOsNq8%2BBmTmMxf%2FA%3D%3D&position=3&pageNum=6&trk=public_jobs_jserp-result_search-card
---------------------------------------------------------------------------------------------------
INFO:li:scraper:('[Unity3D][Worldwide][100]', 'Processed')
2500166312
Unity Developer | CrazyLabs | Skopje, Skopski, North Macedonia | 2021-05-25
https://mk.linkedin.com/jobs/view/unity-developer-at-crazylabs-2500166312?refId=MNbgrdvQ2e4l%2BI7u4dN%2B2g%3D%3D&trackingId=PCMYHKSCI8ZMfEoQIBEp4w%3D%3D&position=4&pageNum=6&trk=public_jobs_jserp-result_search-card
---------------------------------------------------------------------------------------------------
scraper:endThis is a sample of results running the same script in authenticated mode (cookie session):
INFO:li:scraper:('Using strategy AuthenticatedStrategy',)
INFO:li:scraper:('Starting new query', "Query(query=Unity3D options=QueryOptions(limit=100 locations=['Worldwide'] filters=QueryFilters() optimize=False))")
INFO:li:scraper:('[Unity3D][Worldwide]', 'Setting authentication cookie')
INFO:li:scraper:('[Unity3D][Worldwide]', 'Opening https://www.linkedin.com/jobs/search?keywords=Unity3D&location=Worldwide')
INFO:li:scraper:('[Unity3D][Worldwide]', 'Session is valid')
INFO:li:scraper:('[Unity3D][Worldwide]', 'Found 7 jobs')
INFO:li:scraper:('[Unity3D][Worldwide][1]', 'Processed')
2584430881
3D Artist | Grover Gaming | Greenville, NC |
https://www.linkedin.com/jobs/view/2584430881/?eBP=CwEAAAF52zoXWHvsGld9m82V8wVuEAvKb1JlnpRrpQRHHw76tE5GWabqtv77HJSA_hDdA_43rCbrnk5rknkyeHVfvoel-Stt9FtsAlC184Q0JrmIFo1MZ_KpmyluUXhx8mp798XDyxGlr1AvP59oYaCTcrd8iZSj3dr5DyzVXUZainDKL3ZNe7FA-hwehv53jF71-k0V1PhYpJX-aJBEwNREzBNxg6Stf7VPWOdY0yxg8Xthc8Chk4gsV0ceJWTIp1YXzglG1eOwBKsPZFDdNLVWSoLCKgyOQHnPpzKKFfhONc0arbXRMDmpzbO6afenhkY-vku_r_1NrHhht3SnCUACjnNIvRs3nSRzgFEudT8c3B8M-ccqGGSDG_MOEoCr&recommendedFlavor=ACTIVELY_HIRING_COMPANY&refId=WaxI3YP2fZd6bt08pWlqtQ%3D%3D&trackingId=C331G7pwD15BgNHjI8ZMDQ%3D%3D&trk=flagship3_search_srp_jobs
---------------------------------------------------------------------------------------------------
INFO:li:scraper:('[Unity3D][Worldwide][2]', 'Processed')
2576860532
Volunteer Narrative Designer (Remote and No Compensation) | Games for Love | Monroe, WA Remote |
https://www.linkedin.com/jobs/view/2576860532/?eBP=CwEAAAF52zoXWKv6tThhHSgoaQ3x4uPmY84m5Jv1W9_-MzmZQFUTduuIGSOeD15DeR_L54c36yy2D9ybIkGXvjLkFa6IF8mvBnRC4flciV7WING_-JbT6TAgeNghbLVIcLlcUNUfKZqJJOAdfqGl588qwvmQxHgQOEYEaTx7v97Hsv5I6ih_5gewVuFph4CRK6RvRqQGdQAhtiER6e4ly_2DhEvgcgQMEWCgdhU3w1e43_B7ypPxP0yggYYPBc-Qzt1tzVhemhWMcTWjr3lyjIGEPYz4YIj9pgTe1RXA9yjgVBOTbxocQQbnldTmh9ZcbuhNw-oYRVBKaIuAn4O-IFX--KQk0Q&refId=WaxI3YP2fZd6bt08pWlqtQ%3D%3D&trackingId=Wry7KVuxd0TIxW%2FORajIew%3D%3D&trk=flagship3_search_srp_jobs
---------------------------------------------------------------------------------------------------
[...]
INFO:li:scraper:('[Unity3D][Worldwide][98]', 'Processed')
2576855219
Artist / Animator | Scientific Games | Las Vegas, NV |
https://www.linkedin.com/jobs/view/2576855219/?eBP=CwEAAAF52zuZ-iVOOrjGQCzf90Gglw3-rixgaUPdnwcp9DHzxFr3Qp6ncntfjouzxJNm8MSP0BhVKTFfouONEsAJz8R__u2zLR4WFGOACrSwLoSQPbEN97yKIGasZglV3-KAzj4uo-aTqt7-lsxC0ooOc92GZbkCj5Jub1Luy2p0dBxrBzM-M712eK2O-XVbXex6ikrBfRyIkAyu21dGH5xyoarG5mL3_gwz-hKC7l1KmOED5EQYiBCEGLZNfcDOPQ7AySyzvkBGRUzs2XnZeVE_xUAaOBjBjR5YoPlpxlIqwIprpIx6pSEpb7abC2GAsPdrJW3wlG_pXMy4alDnD0Lu2z0UCQ&recommendedFlavor=ACTIVELY_HIRING_COMPANY&refId=HA6PYVHDQliKzy2N89mw8w%3D%3D&trackingId=HGBJqFQhTi8RWo%2BVPFqofg%3D%3D&trk=flagship3_search_srp_jobs
---------------------------------------------------------------------------------------------------
INFO:li:scraper:('[Unity3D][Worldwide][99]', 'Processed')
2577249749
Mid/Senior Software Engineer (C++, Unreal, Game Technology) | Wargaming | Sydney, New South Wales, Australia |
https://www.linkedin.com/jobs/view/2577249749/?eBP=CwEAAAF52zuZ-xwbm3HmMt7ApzusdzA4Tl6w07fB09gpxMT--i4MIoaAM7e01_0I3iy7S0CgpgHLonpuaHA5zIZjLCLP-JyWf0xDijLquLnKIZWI_W3zB6yA5SBYgSI1o8ePwaR12e1-BWB0w1i7qeLPBSrUmE62VHYptg0amVe0LNf_oy6VEDSbuaR6FoZ9b9DXlmdUug3-lpj7l8RBaPQFVGFhuYdxQrwHAOb2hXiDgT1cdhnkgxK3xwISgdZwIsV4wjznHke9tIMq3yEdFQM97Bp5fY_PE5SgkRfa1DP600XIAoklTyUe9MtmJDiPWqEd_ALhz35aDwpGVCk4mrT2FC3_D8GGm4Ka5tS3UbV0nff9wJcVMVN9Ff6Sig&recommendedFlavor=ACTIVELY_HIRING_COMPANY&refId=HA6PYVHDQliKzy2N89mw8w%3D%3D&trackingId=3ayOxrVRx%2BvJCiwSW3ox6w%3D%3D&trk=flagship3_search_srp_jobs
---------------------------------------------------------------------------------------------------
INFO:li:scraper:('[Unity3D][Worldwide][100]', 'Processed')
2580629179
unity高级开发工程师 | 北京兰亭数字科技有限公司 | Hangzhou, Zhejiang, China | 2021-06-03
https://www.linkedin.com/jobs/view/2580629179/?eBP=JOB_SEARCH_ORGANIC&refId=HA6PYVHDQliKzy2N89mw8w%3D%3D&trackingId=IwtKInlI5pVxKT6rPzbGzA%3D%3D&trk=flagship3_search_srp_jobs
---------------------------------------------------------------------------------------------------
scraper:endAre you using the latest version (1.8.4)?
Willempie909
Hi,
Unfortunately the jobs selectors seem to be misaligned.
Underneath an example of what I mean. The collected company name and location do not match the job title and the link.
INFO:li:scraper:('[Unity3D][Worldwide][3]', 'Processed') Senior Unity Engineer Games Factory Talents Worldwide Stockholm, Stockholm County, Sweden 2021-06-04 https://au.linkedin.com/jobs/view/senior-unity-engineer-at-stageverse-2571868917?refId=eM6qeCaakFnm1NoSaL%2B5jg%3D%3D&trackingId=cKutmpDKsnofmyEnY%2BcOrg%3D%3D&position=4&pageNum=0&trk=public_jobs_jserp-result_search-card 2571868917 3738 INFO:li:scraper:('[Unity3D][Worldwide][4]', 'Processed') Infrastructure Analyst Stageverse Worldwide New South Wales, Australia 2021-06-04 https://ca.linkedin.com/jobs/view/infrastructure-analyst-at-robert-half-2582225773?refId=eM6qeCaakFnm1NoSaL%2B5jg%3D%3D&trackingId=qHfJfMitFLKAa2pEQcdN0Q%3D%3D&position=5&pageNum=0&trk=public_jobs_jserp-result_search-card 2582225773 2484 INFO:li:scraper:('[Unity3D][Worldwide][5]', 'Processed') Sound Designer (remote) Robert Half Worldwide Richmond Hill, Ontario, Canada 2021-06-04 https://es.linkedin.com/jobs/view/sound-designer-remote-at-anotherway-2582227069?refId=eM6qeCaakFnm1NoSaL%2B5jg%3D%3D&trackingId=qndYXrwtvQ2HhVFxVRHpnQ%3D%3D&position=6&pageNum=0&trk=public_jobs_jserp-result_search-card 2582227069 * 700