search

ericfourrier / scrape-linkedin

Scrape a public LinkedIn profile.
MIT License
152 stars 51 forks source link

Change User Agent? #8

Open mattFlemingTamr opened 6 years ago

mattFlemingTamr commented 6 years ago

Hi,

I was curious how to change the user agent in conjunction with using this scraper.

-Matt

Andreilys commented 6 years ago

Hey Matt, you can change the user agent when you're initializing CustomRequest, the parameter is list_user_agent.

c = CustomRequest()

c = CustomRequest(list_user_agent=user_agent, list_proxies=list_proxies)

ericfourrier commented 6 years ago

Thanks for your help @Andreilys, @mattFlemingTamr you need to do the following (there is actually a typo in the docs) :

c=CustomRequest(list_user_agent=user_agent, rotate_ua=True, list_proxies=None)

Nevertheless i think it won't be enough to bypass LinkedIn ban even if you pass a GoogleBot user agent or anything similar.

Andreilys commented 6 years ago

Yes seems like the ban is very hard to get around, have you been able to implement anything on your end @ericfourrier ? Couldn't get this to work no matter what I tried (Even Selenium failed). I'm only looking to scrape 50-60 profiles, so not much

ericfourrier commented 6 years ago

I have tried some stuff too the only thing that worked (1 year ago) was buying a pool of residential ips but now it seems even residential ips are quickly banned.