Error: Invalid protocol: null

[cyclops24]

Hi @chill117 ,

I try to fetch some proxy but it seems it return Error: Invalid protocol: null error for many of proxy sources. Is it a bug or I use package in wrong way?

[chill117]

Are you seeing that error when using the proxies or when getting the proxies via this module?

[cyclops24]

I see this when try get proxy by getProxies() function and log it with error event.

[chill117]

I'll have a look at it in the next few days.

[cyclops24]

@chill117 Thanks man. :wink:

[chill117]

I have run the tests a few times at different days/times, but I have not been able to reproduce your issue. It could just be an issue with your ISP or your IP address being blocked by one or more of the proxy sources. Or, if you are using a proxy via the node proxy environment variable, the proxy/ies you are using to get more proxies could be not working properly?

[cyclops24]

Hi @chill117 , First thanks for your attention. Yes I used node proxy environment based on this issue but I think that's not problem because I checked with many proxy with different quality. Also with same node proxy environment each time different sources return Error: Invalid protocol: null that is wrong. @chill117 I used your package as dependency in a Meteor application. If you familiar with Meteor I can create simple test application for more simpler issue reproduce. Tell me if you know Meteor man. :wink:

[chill117]

And what about when you don't use any proxy and you use a VPN instead?

I am familiar with Meteor, but haven't used it for any projects yet. I am not sure if it would cause any problems with this module. You could eliminate Meteor as a suspect if you try using this module outside of Meteor and the problem continues.

[cyclops24]

Hi @chill117, I haven't VPN for test your first suggestion. I create a simple demo app for using proxy-list in Meteor.js see this man: https://github.com/MeteorDemoApps/proxy-lists-demo and specially: https://github.com/MeteorDemoApps/proxy-lists-demo/blob/master/server/main.js

I used freegate as proxy and http://127.0.0.1:8580 in line 13 & 14 is freegate proxy url. It's my output now:

=> Started proxy.                             
=> Started MongoDB.                           
I20160608-13:47:27.815(4.5)? Proxy Received: 24
I20160608-13:47:29.402(4.5)? Proxy Received: 37
W20160608-13:47:29.418(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.419(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.419(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.420(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.420(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.421(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.421(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.421(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.422(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.422(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.422(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.423(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:29.425(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:37.240(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
W20160608-13:47:40.156(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
I20160608-13:47:43.194(4.5)? Proxy Received: 100
W20160608-13:47:45.319(4.5)? (STDERR) Error Occurred: Error: Invalid protocol: null
I20160608-13:47:45.876(4.5)? proxiesCount: 161
=> Started your app.

=> App running at: http://localhost:3000/

[chill117]

You must comment out these lines. Then it will work. You could move them to within the 'end' event callback function and it will work as well.

What is happening there:

• You are setting the HTTP_PROXY and HTTPS_PROXY environment variables to your proxy, but then you are setting them equal to null later in the same code block. Since everything within ProxyLists is asynchronous, your code that sets the HTTP_PROXY and HTTPS_PROXY variables to null runs before any code within ProxyLists.

[cyclops24]

@chill117 Ohhhh man. It is a silly mistake :see_no_evil: I fix that and now my output is: (Just one Error: ETIMEDOUT and other source fetched successfully)

I20160608-14:34:44.923(4.5)? Proxy Fetched: 45
I20160608-14:34:45.066(4.5)? Proxy Fetched: 28
I20160608-14:34:49.844(4.5)? Proxy Fetched: 100
I20160608-14:34:51.160(4.5)? Proxy Fetched: 257
I20160608-14:34:52.950(4.5)? Proxy Fetched: 275
W20160608-14:34:54.598(4.5)? (STDERR) Error Occurred: Error: ETIMEDOUT
I20160608-14:34:57.729(4.5)? Proxy Fetched: 254
I20160608-14:34:58.169(4.5)? Proxy Fetched: 265
I20160608-14:34:59.228(4.5)? Proxy Fetched: 274
I20160608-14:35:00.007(4.5)? Proxy Fetched: 276
I20160608-14:35:00.452(4.5)? Proxy Fetched: 273
I20160608-14:35:04.968(4.5)? Proxy Fetched: 278
I20160608-14:35:06.570(4.5)? Proxy Fetched: 274
I20160608-14:35:07.021(4.5)? Proxy Fetched: 273
I20160608-14:35:10.011(4.5)? Proxy Fetched: 0
I20160608-14:35:10.878(4.5)? Proxy Fetched: 268
I20160608-14:35:11.315(4.5)? Proxy Fetched: 269
I20160608-14:35:13.322(4.5)? Proxy Fetched: 0
I20160608-14:35:17.880(4.5)? Proxy Fetched: 281
I20160608-14:35:22.531(4.5)? Total proxy count: 3690

@chill117 is it right way to unset node proxy environment variable?

process.env.HTTP_PROXY = null;
process.env.HTTPS_PROXY = null;

[chill117]

Ohhhh man. It is a silly mistake.

It happens :)

I fix that and now my output is: (Just one Error: ETIMEDOUT and other source fetched successfully)

Occasional timeout errors are expected. Some of the proxy sources can have down-time or be overloaded or there could be other connectivity issues.

is it right way to unset node proxy environment variable?

Yes, that is the preferred method of un-setting keys on objects in JavaScript. But from seeing the previous error you were getting (Invalid protocol: null), I think the request module is treating a null value as if it is set and is trying to use it as a proxy. So you might have to set those environment variables equal to undefined (which is a bad practice) or use delete process.env.HTTP_PROXY.

[cyclops24]

@chill117 really thanks for your attention. :heart: I also done some clean up on code and fix issues (also add delete process.env.HTTP_PROXY) and add some readme. You can use it repo url on proxy-list readme if you think it's useful for meteor users :wink:

[cyclops24]

@chill117 I close this issue man. Again thanks :wink: