chill117 / proxy-verifier

NodeJS module to check proxies: if functional, anonymity level, tunneling, supported protocols.
MIT License
76 stars 16 forks source link

use test as sync function #18

Closed mnlbox closed 7 years ago

mnlbox commented 7 years ago

Hi @chill117 , I try using this library for check some proxies but after some check it's return Maximum number of open connections reached. message. How we can use test function as sync function. Can you help me?

chill117 commented 7 years ago

When you test a single proxy, the most number of connections at once would be 6 or so. Are you testing all of your proxies in parallel?

mnlbox commented 7 years ago

No I use a foreach for check a proxy list. How we can increase connection number or check this in parallel?

chill117 commented 7 years ago

Are you doing something like this?

// !! Do not use this code !!
for (var proxy in proxies) {
    ProxyVerifier.testAll(proxy, function(error, result) {
    });
}

Because this would lead to the issue you're having. The problem with the above code is that the loop starts the testing for each proxy without waiting for the previous tests to finish. So you will end up with ~6 simultaneously open HTTP requests per proxy.

To run the tests in series (instead of parallel), try something like this:

var async = require('async');

async.mapSeries(proxies, function(proxy, next) {
    ProxyVerifier.testAll(proxy, next);
}, function(error, results) {
    // This callback executes after all the proxies have been checked.
});