yatish27 / linkedin-scraper

Scrapes the public profile of the linkedin page
MIT License
551 stars 221 forks source link

Mechanize::ResponseCodeError: 999 => for -- https://www.linkedin.com/in/jeffweiner08 #93

Open washingon opened 8 years ago

washingon commented 8 years ago

linkedin-scraper (2.1.1)

irb(main):001:0> require 'linkedin-scraper' => true irb(main):002:0> profile = Linkedin::Profile.new("http://www.linkedin.com/in/jeffweiner08") Mechanize::ResponseCodeError: 999 => for -- https://www.linkedin.com/in/jeffweiner08

pinymax commented 8 years ago

Same for me. Works well on local machine, but doesn't work on Heroku or AWS. Is any way to use proxy or somehow fix this problem?

natemcguire commented 8 years ago

Same here. Looks like we need to set the user agent for mechanize.

https://github.com/sparklemotion/mechanize/issues/336

pinymax commented 8 years ago

@natemcguire "The latest version uses randomuser agent" So I guess, it wouldn't ihelp

CodeCompiler commented 7 years ago

Need to put a proxy service like storm proxies into this so the IP resolution doesn't come back

yatish27 commented 7 years ago

@CodeCompiler Any suggestion PRs

chamnap commented 7 years ago

@CodeCompiler, does the storm proxies work for you? I used myprivateproxy, but it doesn't work at all.

rcavezza commented 7 years ago

I just bought 10 proxies from instantproxies and I can't get a single valid response, whereas I can get a few responses per day without using a proxy. I'm only passing proxy_ip and proxy_port, not username and password.

rcavezza commented 7 years ago

I also bought a few IPs from StormProxies and I am having the same issues. I'm now using username and password on localhost - same result, 999 Mechanize error.

rcavezza commented 7 years ago

I created a PR, but it didn't fix my issue. If anyone has any insight into why this is occurring for me and can point me in the right direction on how to fix it, I'd really appreciate it. I can't tell if this is happening for everyone, or a small subset of people (based on the previous comments).

rcavezza commented 7 years ago

I have not been able to get over this issue. I'm going to attempt to use the mechanize gem without linkedin-scraper in hopes that I will be able to get it to work correctly. If I find a solution, I will come back and apply the fix to this repo.

iliasnaamane commented 7 years ago

Same error here /home/youpi/.rvm/gems/ruby-2.2.0/gems/mechanize-2.7.5/lib/mechanize/http/agent.rb:949:in `response_read': 999 Kindly help me, thanks

Startouf commented 7 years ago

Hey, any updates on this ? Also it seems like a duplicate of https://github.com/yatish27/linkedin-scraper/issues/70

I have tried StormProxy, limeProxies instantProxies and all IPs were already blocked apparently, even those who were said to be "linkedin proxies"

matsossah commented 7 years ago

@iliasnaamane @rcavezza @chamnap @yatish27 @CodeCompiler @pinymax @natemcguire @washingon Has anybody been able to resolve this issue? @Startouf and I work together and haven't found a single solution yet :/ Thanks in advance for any help you could provide!

borenstejn commented 7 years ago

Hi Guys,

I'm just trying to use the gem locally and after having succeeded twice, I end up with the same message: Mechanize::ResponseCodeError: 999 => for -- http://www.linkedin.com/in/jeffweiner08

I thought it was a problem with my IP address so I used a VPN (betternet) to trick LinkedIn, but i still have the same error coming. Do you have any clue for me ?

Thanks a lot

Startouf commented 7 years ago

Hey @borenstejn I actually got a mail from a Proxy company that explained to me it was very hard nowadays to get a hold of proxys that work with Linkedin since they've become very aggressive.

FYI list of proxies already tried

I'd be glad to have your feedback in case you find something that works, thank you very much.

georgenicholas commented 6 years ago

Did anyone ever figure this out? I'm getting the 999 error, it hasn't worked even once yet. I'm running this locally.

shantilal-narola commented 6 years ago

Any update on this?

tp00012x commented 5 years ago

Has somebody figured this out?

sakinho commented 4 years ago

This error happens to me locally with no proxy. Any idea of how to solve it?