johntitus / node-horseman

Run PhantomJS from Node
MIT License
1.45k stars 124 forks source link

new Horseman ({proxy}) failed to get url #211

Open delormemarie240 opened 8 years ago

delormemarie240 commented 8 years ago

Hello, Without proxy options in new Horseman, script is working great. But when I run this script I get this error: "Unhandled rejection Error: Failed to GET url: http://www.check2ip.com". When I turn proxyType as 'none' I access to check2ip but IP is not 142.91.204.5

var Horseman = require("node-horseman");

var horseman = new Horseman({
proxy: '142.91.204.5:22',

 proxyType: 'socks5',

 proxyAuth: 'root:msn43w1'

});

horseman
    .open('http://www.check2ip.com')
    .screenshot('check2ip.jpg')
    .log()
    .close();
awlayton commented 8 years ago

Are you sure you have the right username, password, and proxy type? It seems like you're failing to connect to the proxy.

delormemarie240 commented 8 years ago

yes I use right user name, pw and proxy type. I can connect putty or bitvise with it.

awlayton commented 8 years ago

Can you post the output from running it with DEBUG='horseman' and BLUEBIRD_DEBUG=1 set?

aryeharmon commented 8 years ago

I im having this issue as well.

horseman .setup() creating phantom instance 21 +267ms
Trying to get: https://xxxxxxxxxxxxxxxxxxxxxxx, with proxy: xx.xx.xx.xx:xxxx
horseman phantom created +53ms
horseman phantom version 2.1.1 +7ms
horseman page created +6ms
horseman phantomjs onLoadFinished triggered +5ms success NaN
horseman phantom created +19ms
horseman injected jQuery +2ms
horseman phantom version 2.1.1 +1ms
horseman page created +6ms
horseman phantomjs onLoadFinished triggered +7ms success NaN
horseman injected bluebird +4ms
horseman phantom created +5ms
horseman phantom version 2.1.1 +3ms
horseman .userAgent() set +4ms Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
horseman page created +1ms
horseman .open() +0ms https://xxxxxxxxxxxxxxxxxxxxxxx
horseman injected jQuery +3ms
horseman phantomjs onLoadFinished triggered +4ms success NaN
horseman injected bluebird +14ms
horseman injected jQuery +2ms
horseman .userAgent() set +4ms Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
horseman .open() +1ms https://xxxxxxxxxxxxxxxxxxxxxxx
horseman injected bluebird +7ms
horseman .userAgent() set +5ms Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
horseman .open() +0ms https://xxxxxxxxxxxxxxxxxxxxxxx
horseman phantom created +116ms
horseman phantom version 2.1.1 +3ms
horseman page created +4ms
horseman phantomjs onLoadFinished triggered +5ms success NaN
horseman injected jQuery +11ms
horseman injected bluebird +11ms
horseman .userAgent() set +7ms Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
horseman .open() +0ms https://xxxxxxxxxxxxxxxxxxxxxxx
horseman phantomjs onLoadFinished triggered +376ms fail 1
[Error: Failed to GET url: https://xxxxxxxxxxxxxxxxxxxxxxx

when i access it without a proxy everything works, this seems to happen on https sites but not all.

notice the horseman phantomjs onLoadFinished triggered +4ms success and then horseman phantomjs onLoadFinished triggered +376ms fail 1

aryeharmon commented 8 years ago

I did a test and sites that only are available over https this always happens.

awlayton commented 8 years ago

Why is your proxy running on the SSH port @delormemarie240? Also, that IP does not seem to be up...

awlayton commented 8 years ago

Can you connect to HTTPS sites _without_ a proxy @aryeharmon?

delormemarie240 commented 8 years ago

1)out put : horseman using PhantomJS from phantomjs-prebuilt module +0ms horseman .setup() creating phantom instance 1 +6ms horseman phantom created +180ms horseman phantom version 2.1.1 +15ms horseman page created +7ms horseman phantomjs onLoadFinished triggered +6ms success NaN horseman injected jQuery +14ms horseman .open() +3ms https://www.check2ip.com/ horseman phantomjs onLoadFinished triggered +54ms fail 1 horseman .close(). +5ms Unhandled rejection Error: Failed to GET url: https://www.check2ip.com/

2) I run proxy on port 22 by default as it is working on bitvise or putty. Which proxy should I used ? 80 isn't working and 1080 access to Apache webpage server.

3) I give you rights proxy informations : proxy: '142.91.104.1:80', proxyAuth: 'root:flipiflop'

awlayton commented 8 years ago

80 is a weird port as well @delormemarie240 , that's HTTP. In fact, there's a web server running on 80. If you're just picking random ports use high numbers (like in the 1000's or higher).

I do not know anything about using bitvise or putty. Are you running the proxy on the machine that is running horseman?

delormemarie240 commented 8 years ago

Yes I run proxy on the machine that is running horseman. As I use pm2 (process monitoring) I would like to - from one proxy - start many horseman instances on others proxies. It would be faster than start horseman instance from each proxy one by one ... Could you suggest to me which hosting supplier provides working proxy on horseman?

awlayton commented 8 years ago

Horseman should work with any SOCKSv5 proxy. Right now my only guess as to what could be happening is either:

  1. Your proxy is not setup the way you think, or
  2. there is a problem with proxies in PhatnomJS.

I do not see how the horseman code could be causing this (it just passes the proxy options right to phantom), and I think 2 is _very_ unlikely.

johntitus commented 8 years ago

@delormemarie240, can you try setting the horseman timeout to something big, like 30 seconds, and retrying this?

spacedevin commented 6 years ago

had the same issue and was just a timeout issue. when I did .waitForSelector('body')before everything else it worked fine.