Add custom URLs check

[cyclops24]

Hi @chill117, Is it possible to verify proxy based on specific URL? For example check list for google-passed proxy or other specific URL.

[chill117]

Yes, but in a little bit of a hacky way. If you see these lines:

    _protocolTestUrl: 'http://bitproxies.eu/api/v2/check',
    _anonymityTestUrl: 'http://bitproxies.eu/api/v2/check',
    _tunnelTestUrl: 'https://bitproxies.eu/api/v2/check'

You can override those test URLs with your own custom URL. But you will need to make sure that the response from your custom URL (the HTTP status code and data) satisfies this check:

    reachedProxyCheckService: function(data, status, headers) {

        return status === 200 &&
                _.isObject(data) &&
                _.has(data, 'ipAddress') &&
                _.has(data, 'headers');
    }

Or you could override that method too, and have your own custom check.

[cyclops24]

Thanks @chill117, Did you know any check service for google check?

[chill117]

@cyclops24 What do you mean by "google check"?

[cyclops24]

I want to check proxy for google blocked or not to create a google passed list. Many proxy verified with this npm package as ok but blocked by google. I need to check block state for google or other sites.

[chill117]

We could add in a google check to this module. Would need to know exactly what a "blocked" request to google looks like (HTTP status code, body, headers, etc.).

[cyclops24]

@chill117 I search a little and try to provide some information about this. Please see this: http://ninjaseotools.com/bulk-proxy-checker.php maybe useful for this feature idea.

[cyclops24]

Hi @chill117 , I searched a lot about this feature and this is my result:

• I didn't find any web service for check google proxy.
• Some code that worked as a google proxy checker (in Python language) use this criteria for detect blocked or not working proxy:

def check(host, port):
    try:
        proxy_handler = urllib2.ProxyHandler({'http': str(host)+":"+str(port)})
        opener = urllib2.build_opener(proxy_handler)
        opener.addheaders = [('User-agent', 'Mozilla/5.0')]
        urllib2.install_opener(opener)
        req=urllib2.Request('https://www.google.com/#q=test')  # change the URL to test here
        sock=urllib2.urlopen(req)
    except urllib2.HTTPError, e:
    with open('blocked.txt', 'a') as file:
      file.write(host+":"+port+"\n")
        return False
    except Exception, detail:
    with open('blocked.txt', 'a') as file:
      file.write(host+":"+port+"\n")
        print "Bad Proxy - "+host+":"+port
        return False
    with open('check.txt', 'a') as file:
      file.write(host+":"+port+"\n")
    return True

as you see above it's send a request to a search page on google like https://www.google.com/#q=test and then if HTTPError or Exception exist it return false and otherwise it return true.

• Some other code that worked as a google proxy checker (again in Python language) use another criteria for detect blocked or not working proxy:

try:
                proxy_handler = urllib2.ProxyHandler({'http':proxy_info})
                opener = urllib2.build_opener(proxy_handler)
                opener.addheaders = [('User-agent','Mozilla/5.0')]
                urllib2.install_opener(opener)
                req = urllib2.Request("http://www.google.com")
                sock=urllib2.urlopen(req, timeout= 7)
                rs = sock.read(1000)
                if '<title>Google</title>' in rs:
            with open('passed.txt', 'a') as file:
                file.write(host+":"+port+"\n")
                    output.append(('0',proxy_info))
                else:
                    raise "Not Google"
            except:
                output.append(('x',proxy_info))

This approach send a request to http://www.google.com and check that <title>Google</title> exist in response or not. If exist proxy is correct otherwise proxy not working or blocked.

I think second approach (check HTTPError or Exception) is better and more general for example if we want to check proxy based on a custom URL we can use a function like this:

var ProxyVerifier = require('proxy-verifier');

var proxy = {
    ipAddress: '127.0.0.1',
    port: 8080,
    protocols: ['http', 'https'],
    checkUrls: ["https://www.google.com/#q=test", "https://www.facebook.com", otherCustomURL, ...]
};

// this function check acess to checkUrls with http and https and return ok if it's
ProxyVerifier.testAll(proxy, function(error, results) {

    if (error) {
        // Some unusual error occurred.
    } else {
        // The results object contains a result object for each protocol.
    }
});

And a result like this:

{
    anonymityLevel: 'elite',
    protocols: {
        {
    http: {
        ok: false,
        error: {
            message: 'socket hang up',
            code: 'ECONNRESET'
        }
    },
    https: {
        ok: false,
        error: {
            message: 'socket hang up',
            code: 'ECONNRESET'
        }
    }
}
    },
    checkUrls: [
        { url: "https://www.google.com/#q=test", protocol: 'http', ok: true },
        { url: "https://www.google.com/#q=test", protocol: 'https', ok: true },
        { url: "https://www.facebook.com", protocol: 'http', ok: true },
        { url: "https://www.facebook.com", protocol: 'https', ok: true },
        { url: otherCustomURL, protocol: 'http', ok: true },
        { url: otherCustomURL, protocol: 'https', ok: true }
    ],
    tunnel: {
        ok: true
    },
    country: 'cz'
}

So I think it''s better to rename this issue to something like Verify Custom URL or something like this and with this approach user can do a simple check that is a specific URL is accessible from a specific proxy in specific protocol or not.

[cyclops24]

@chill117, you can also see this and this for some idea. This two packages also support regex lookup in response. I think mix this feature with your proxy-verifier has awesome result. :wink:

[chill117]

Sounds good. Thanks for the input!

[chill117]

Done by #17

[cyclops24]

@chill117 It's so good man. Thanks. 😉 I'm going to test it.