Closed hexdex22 closed 3 years ago
kboghe
commented
3 years ago Dear Raj,
Thank you so much for your kind words and for reporting this issue. I shall implement a fix asap - hopefully today - and I'll report back to you when there's a new version available on pypi.
Best, Kristof
kboghe
commented
3 years ago Dear Raj, I updated the package on Pypi (0.2.6). I implemented a different and more flexible solution than the one you suggested, but the main idea is the same: the script saves your original ip now when you're disconnected from any VPN server (in the initialize_VPN function), saves this original ip for later usage in the rotate_VPN function by writing it into the instructions dict, and checks whether the new ip is not the same as the previous (e.g. a previous NordVPN server) or original ip in the rotate_VPN function. Could you update the package version and report whether the error is resolved or not? PS: here's a tip to make server rotation truly random: instead of using the 'random countries 60' option, type in 'complete rotation' in the area_input parameter instead. It's a new feature I implemented in version 0.2.5 onwards! Thanks again for reporting this issue,Kristof
hexdex22
commented
3 years ago Hi Kristof,
Good news - after 500 rotations in one trial and various small trials there were no leaks. The software behaved perfectly.
I think your implementation by saving the real IP is a good idea, but I do feel there is some value in returning some form of a result. It would be nice to get the IP, country and server number. The IP can be used to maintain a local list of blacklisted IPs that you want to avoid:
while True:
result = rotate_vpn()
if result.ip not in blacklisted_ip: breakThe main reason I would like the IP is that, even after all this testing, I'm still a bit paranoid that some freak situation will result in the real IP being returned :)
Best regards,
Raj
kboghe
commented
3 years ago Hi Raj,
Well, I'm not a 100% sure what you're getting at to be honest, but in any case:
1) There is a list of blacklisted ip's at each rotation: the original ip and the previous ip. In the first rotation, both blacklisted ip's are the same of course, but after the first rotation the list includes the original ip (recorded when the program disconnects you from NordVPN servers - if necessary - during the initialize_VPN phase) and whatever NordVPN server you previously connected to. It is therefore - in theory anyway - impossible that NordVPN switcher will simply continue using your original ip. The script will break if the ip keeps being unchanged to either the previous ip or original ip after a rotation attempt, similar to the script snippet you suggest here.
2) However, I do agree that it would be useful to extend this list of blacklisted ip's so that the script will not connect to the same NordVPN server in a relatively short time span. The odds that this happens is still extremely low, though. If you use complete rotation, the script picks one random option out of 4000+ available servers, which means the probability of connecting to the same server twice during 3 rotations is 1/4000+ . For example, logging the last 500 servers in a separate list and adding it to the blacklisted ip's seems reasonable.
I'll keep this in mind for the next update, but I rather keep version 0.2.6 up and running for a while now before performing another update and overwhelming users with update after update. :)
Thanks again for reporting this issue and rest assured: the script will break if your real ip keeps popping up. If you experience any other issues and/or you have any further suggestions for improvement, do not hesitate to contact me.
hexdex22
commented
3 years ago Hi,
Sorry. Further overnight testing revealed two leaks. Maybe, an interruption of the internet service.
The rotation numbers are 26 and 48 which means they didn't occur as a single event.
Best regards,
Raj
26 2021-03-07 06:23:12.572381 82.30.117.28 48 2021-03-07 06:40:03.866818 82.30.117.28
from sys import exit import beepy as beep import pandas as pd import requests import requests_cache from bs4 import BeautifulSoup from pprint import pprint
from datetime import datetime
import time
from nordvpn_switcher import initialize_VPN, rotate_VPN, terminate_VPN
vpn_instructions = initialize_VPN(area_input=["random countries 60"])
old_ip = "" for i in range(1000): st = time.time() res = rotate_VPN(vpn_instructions) print("Rotation time:", time.time() - st) ip_add = requests.get("http://uklcds.com/check_ip.php").text print(i, res, ip_add) if ip_add == "82.30.117.28": with open("vnp_leak.txt", "a+") as f: f.write(str(i) + " " + str(datetime.now()) + " " + ip_add + "\n")
Connecting you to Chile ...
your new ip-address is: 45.228.209.147
Done! Enjoy your new server.
Rotation time: 22.921218633651733 84 None 45.228.209.147
Your current ip-address is: 45.228.209.147
Connecting you to Brazil ...
your new ip-address is: 196.240.255.12
Done! Enjoy your new server.
Rotation time: 22.42352867126465 Traceback (most recent call last):
File "C:\Users\X\anaconda3\lib\site-packages\urllib3\connection.py", line 169, in _new_conn conn = connection.create_connection(
File "C:\Users\X\anaconda3\lib\site-packages\urllib3\util\connection.py", line 73, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "C:\Users\X\anaconda3\lib\socket.py", line 918, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
gaierror: [Errno 11002] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\X\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen httplib_response = self._make_request(
File "C:\Users\X\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request conn.request(method, url, **httplib_request_kw)
File "C:\Users\X\anaconda3\lib\site-packages\urllib3\connection.py", line 234, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\X\anaconda3\lib\http\client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\X\anaconda3\lib\http\client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\X\anaconda3\lib\http\client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\X\anaconda3\lib\http\client.py", line 1010, in _send_output self.send(msg)
File "C:\Users\X\anaconda3\lib\http\client.py", line 950, in send self.connect()
File "C:\Users\X\anaconda3\lib\site-packages\urllib3\connection.py", line 200, in connect conn = self._new_conn()
File "C:\Users\X\anaconda3\lib\site-packages\urllib3\connection.py", line 181, in _new_conn raise NewConnectionError(
NewConnectionError: <urllib3.connection.HTTPConnection object at 0x0000019B23D7F850>: Failed to establish a new connection: [Errno 11002] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\X\anaconda3\lib\site-packages\requests\adapters.py", line 439, in send resp = conn.urlopen(
File "C:\Users\X\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen retries = retries.increment(
File "C:\Users\X\anaconda3\lib\site-packages\urllib3\util\retry.py", line 573, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause))
MaxRetryError: HTTPConnectionPool(host='uklcds.com', port=80): Max retries exceeded with url: /check_ip.php (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000019B23D7F850>: Failed to establish a new connection: [Errno 11002] getaddrinfo failed'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\X.spyder-py3\carpages.py", line 27, in
File "C:\Users\X\anaconda3\lib\site-packages\requests\api.py", line 76, in get return request('get', url, params=params, **kwargs)
File "C:\Users\X\anaconda3\lib\site-packages\requests\api.py", line 61, in request return session.request(method=method, url=url, **kwargs)
File "C:\Users\X\anaconda3\lib\site-packages\requests\sessions.py", line 542, in request resp = self.send(prep, **send_kwargs)
File "C:\Users\X\anaconda3\lib\site-packages\requests\sessions.py", line 655, in send r = adapter.send(request, **kwargs)
File "C:\Users\X\anaconda3\lib\site-packages\requests\adapters.py", line 516, in send raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='uklcds.com', port=80): Max retries exceeded with url: /check_ip.php (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000019B23D7F850>: Failed to establish a new connection: [Errno 11002] getaddrinfo failed'))
System Information report written at: 03/07/21 12:25:01 System Name: SERVER [System Summary] Item Value OS Name Microsoft Windows 10 Pro Version 10.0.19042 Build 19042 Other OS Description Not Available OS Manufacturer Microsoft Corporation System Name SERVER System Manufacturer To Be Filled By O.E.M. System Model To Be Filled By O.E.M. System Type x64-based PC System SKU To Be Filled By O.E.M. Processor Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz, 4001 Mhz, 4 Core(s), 8 Logical Processor(s) BIOS Version/Date American Megatrends Inc. P2.10, 13/10/2015 SMBIOS Version 2.8 Embedded Controller Version 255.255 BIOS Mode Legacy BaseBoard Manufacturer ASRock BaseBoard Product Z97 Pro3 BaseBoard Version Platform Role Desktop Secure Boot State Unsupported PCR7 Configuration Binding Not Possible Windows Directory C:\Windows System Directory C:\Windows\system32 Boot Device \Device\HarddiskVolume1 Locale United Kingdom Hardware Abstraction Layer Version = "10.0.19041.488" Username SERVER\X Time Zone GMT Standard Time Installed Physical Memory (RAM) 16.0 GB Total Physical Memory 15.4 GB Available Physical Memory 10.4 GB Total Virtual Memory 17.8 GB Available Virtual Memory 12.2 GB Page File Space 2.38 GB Page File C:\pagefile.sys Kernel DMA Protection Off Virtualisation-based security Not enabled Device Encryption Support Reasons for failed automatic device encryption: TPM is not usable, PC R7 binding is not supported, Hardware Security Test Interface failed and the device is not Modern Standby, Un-allowed DMA-capable bus/device(s) detected, TPM is not usable Hyper-V - VM Monitor Mode Extensions Yes Hyper-V - Second Level Address Translation Extensions Yes Hyper-V - Virtualisation Enabled in Firmware Yes Hyper-V - Data Execution Protection Yes
kboghe
commented
3 years ago Hi Raj,
Huh, that's nothing less than inexplicable to me how the ip leakeage could possibly occur. The script should simpyly throw an error when your ip remains unchanged. I'll look into this issue later this week and find out what exactly is going on here.
Anyway, I am at least satisfied with the fact that the package switches you to a different ip in 99+% of the cases. For the occasional hickup, you can always build an additional safety net into your own script ( as in: if new ip = original ip, rotate the servers again).
Thanks again for bringing this issue to my attention and I do hope you'll continue to find nordvpn-switcher a useful tool for your webscraping projects. :)
Hi,
I love the way your software works and it's exactly what I need for scraping projects.
Unfortunately, there is an issue with it sometimes using the real IP address leaving us venerable to be discovered as the scraper.
Sometimes the VPN does not provide an IP address and we get a message:
Your current ip-address is: 37.120.221.188 Connecting you to North Macedonia ... your new ip-address is: 82.30.151.28 <<== NOT VPN ADDRESS Done! Enjoy your new server.
But 82.30.151.28 is my real IP address.
Frequency of this error: 6 2021-03-05 11:47:34.072958 82.30.151.28 13 2021-03-05 11:51:22.841955 82.30.151.28 15 2021-03-05 11:53:05.327267 82.30.151.28 22 2021-03-05 11:56:45.462404 82.30.151.28 29 2021-03-05 12:00:54.673857 82.30.151.28 38 2021-03-05 12:07:33.343003 82.30.151.28 45 2021-03-05 12:11:04.391940 82.30.151.28 51 2021-03-05 12:15:23.405243 82.30.151.28 58 2021-03-05 12:19:18.352110 82.30.151.28 65 2021-03-05 12:22:52.992605 82.30.151.28 72 2021-03-05 12:26:18.048769 82.30.151.28 79 2021-03-05 12:30:05.551850 82.30.151.28 86 2021-03-05 12:33:14.990266 82.30.151.28
** I stopped the program after 90 tries.
Method used to check:
-- coding: utf-8 --
get Name, Price, Colour for first 10-15 pages
from sys import exit import pandas as pd import requests import requests_cache from bs4 import BeautifulSoup from pprint import pprint from datetime import datetime
import time
from nordvpn_switcher import initialize_VPN, rotate_VPN, terminate_VPN
vpn_instructions = initialize_VPN(area_input=["random countries 60"])
old_ip = "" for i in range(500): res = rotate_VPN(vpn_instructions) ip_add = requests.get("http://uklcds.com/check_ip.php").text <-- just a little script on my server which returns the requester IP print(i, res, ip_add) if ip_add == "82.30.151.28": with open("vnp_leak.txt", "a+") as f: f.write(str(i) + " " + str(datetime.now()) + " " + ip_add + "\n")
Suggestion:
rotate_VPN() returns the new ip address so we can wait until a good vpn address is available
while rotate_vpn() == my_real_ip: time.sleep(10)
Best regards,
Raj