--proxy not working if country-check failed

PerchunPak commented 2 years ago

Config

{
    "jobs":[
      {
        "type":"http",
        "count":5,
        "args":{
          "request":{
            "method":"GET",
            "path":"https://iplogger.org/*******"
          },
          "client":{
            "static_host":{
              "addr":"148.251.234.83:443",
              "is_tls":"true"
            }
          }
        }
      }
    ]
  }

Steps to Reproduce the Problem

Copy config.
Change ip-grabber link to yours.
Run programm with command docker run -d --name db1000n-test -v **ABSOLUTE PATH TO FOLDER WITH CONFIG**:/config ghcr.io/arriven/db1000n --proxy "{{ join (split (get_url \"https://raw.githubusercontent.com/porthole-ascend-cinnamon/proxy_scraper/main/proxies.txt\") \"\n\") \",\" }}" -c /config/config.json.
Check site with statistic, result is

Untitled

Logs

2022/04/04 19:22:38.181987 main.go:51: DB1000n [Version: v0.8.17][PID=1]
2022/04/04 19:22:38.300832 countrychecker.go:57: Checking IP address, attempt #1
2022/04/04 19:22:38.300955 countrychecker.go:107: proxy config detected, using it to check country
2022/04/04 19:22:38.301481 countrychecker.go:113: using proxy socks5://123.150.95.142:7302
2022/04/04 19:22:41.302968 countrychecker.go:125: Can't check users country. Please manually check that VPN is enabled or that you have non Ukrainian IP address.
2022/04/04 19:22:41.303121 countrychecker.go:57: Checking IP address, attempt #2
2022/04/04 19:22:41.303166 countrychecker.go:107: proxy config detected, using it to check country
2022/04/04 19:22:41.304303 countrychecker.go:113: using proxy http://23.81.127.35:8118
2022/04/04 19:22:44.303361 countrychecker.go:125: Can't check users country. Please manually check that VPN is enabled or that you have non Ukrainian IP address.
2022/04/04 19:22:44.304577 countrychecker.go:57: Checking IP address, attempt #3
2022/04/04 19:22:44.304650 countrychecker.go:107: proxy config detected, using it to check country
2022/04/04 19:22:44.305342 countrychecker.go:113: using proxy socks5://175.155.137.170:28643
2022/04/04 19:22:47.305521 countrychecker.go:125: Can't check users country. Please manually check that VPN is enabled or that you have non Ukrainian IP address.
2022/04/04 19:22:47.318343 config.go:74: Loading config from "/config/config.json"
2022/04/04 19:22:47.318671 runner.go:103: New config received, applying
2022/04/04 19:22:47.318793 runner.go:208: 5 job instances (re)started
2022/04/04 19:22:47.668191 http.go:147: Attacking https://iplogger.org/*******
2022/04/04 19:22:47.669559 http.go:147: Attacking https://iplogger.org/*******
2022/04/04 19:22:47.675714 http.go:147: Attacking https://iplogger.org/*******
2022/04/04 19:22:47.676897 http.go:147: Attacking https://iplogger.org/*******
2022/04/04 19:22:47.680904 http.go:147: Attacking https://iplogger.org/*******
2022/04/04 19:22:54.821979 main.go:137: Terminating

Specifications

Version: db1000n v0.8.17
Platform: ghcr.io/arriven/db1000n
Subsystem: Docker

PerchunPak commented 2 years ago

Command for docker used in Windows system. I don't know if it will work on Linux, so here tested command only for Linux: docker run -d --name db1000n-test -v **ABSOLUTE PATH TO FOLDER WITH CONFIG**:/config ghcr.io/arriven/db1000n --proxy '{{ join (split (get_url "https://raw.githubusercontent.com/porthole-ascend-cinnamon/proxy_scraper/main/proxies.txt") "\n") "," }}' -c /config/config.json

PerchunPak commented 2 years ago

Could be caused by:

you might need a lot of retries for countrychecker to find working proxy (default is 3 but I didn't manage to get a result with this list even with 15 retries)

In this discussion, also from this comment:

The app won't crash and continue working even if it doesn't find working proxy

Fix will be to add this arguments in CLI:

--proxy-retry: Retry n times to find working proxy, can be also always.
--proxy-crash: Will crash app, when there will no finded working proxy. If in --proxy-retry will be always, crash only if all proxy scanned. Can't be enabled if --proxy-cycle enabled too. (Default: false)
--proxy-cycle: If there is --proxy-retry=always, this argument will force to start country check again. Can't be enabled if --proxy-crash enabled too. (Default: true)

arriven commented 2 years ago

I found what the actual underlying issue is: current implementation supports only socks5 proxies and falls back to direct connection for other protocols. I've only used socks5 proxies for my testing so never found it out, will fix shortly

arriven commented 2 years ago

fixed falling back to direct connection in https://github.com/Arriven/db1000n/commit/36af0e5488ea2e4f962bd2601d7777facbd13760

looking on how to make it accept http and socks4 proxies as well

arriven commented 2 years ago

fixed in v0.8.18 (being built right now)

PerchunPak commented 2 years ago

Something with 0.8.18 really wrong. Command docker run --rm --name db1000n ghcr.io/arriven/db1000n --proxy "{{ join (split (get_url \"https://raw.githubusercontent.com/porthole-ascend-cinnamon/proxy_scraper/main/proxies.txt\") \"\n\") \",\" }}" > log.txt Output: log.txt

arriven commented 2 years ago

what exactly is wrong in that log? or you mean that it takes long time to output targets?

arriven commented 2 years ago

ah, I see, you didn't redirect stderr to the log

arriven commented 2 years ago

0.8.19 with the fix incoming (I forgot that map is a reference type in go)

PerchunPak commented 2 years ago

Seems like it force to use third proxy, even if it not working.

2022/04/05 17:01:13.088470 main.go:51: DB1000n [Version: v0.8.19][PID=1]
2022/04/05 17:01:13.190811 countrychecker.go:57: Checking IP address, attempt #1
2022/04/05 17:01:13.190910 countrychecker.go:107: proxy config detected, using it to check country
2022/04/05 17:01:13.191610 countrychecker.go:113: using proxy socks4://185.14.149.178:4153
2022/04/05 17:01:14.244039 countrychecker.go:125: Can't check users country. Please manually check that VPN is enabled or that you have non Ukrainian IP address.
2022/04/05 17:01:14.244159 countrychecker.go:57: Checking IP address, attempt #2
2022/04/05 17:01:14.244194 countrychecker.go:107: proxy config detected, using it to check country
2022/04/05 17:01:14.244914 countrychecker.go:113: using proxy http://45.153.216.202:8259
2022/04/05 17:01:14.425362 countrychecker.go:125: Can't check users country. Please manually check that VPN is enabled or that you have non Ukrainian IP address.
2022/04/05 17:01:14.426621 countrychecker.go:57: Checking IP address, attempt #3
2022/04/05 17:01:14.426763 countrychecker.go:107: proxy config detected, using it to check country
2022/04/05 17:01:14.427452 countrychecker.go:113: using proxy socks4://45.67.229.104:30003
2022/04/05 17:01:17.427934 countrychecker.go:125: Can't check users country. Please manually check that VPN is enabled or that you have non Ukrainian IP address.
2022/04/05 17:01:17.441904 config.go:74: Loading config from "/config/config.json"
2022/04/05 17:01:17.442259 runner.go:103: New config received, applying
2022/04/05 17:01:17.442394 runner.go:208: 5 job instances (re)started
2022/04/05 17:01:17.775237 http.go:147: Attacking https://iplogger.org/*******
2022/04/05 17:01:17.794898 http.go:147: Attacking https://iplogger.org/*******
2022/04/05 17:01:17.804509 http.go:147: Attacking https://iplogger.org/*******
2022/04/05 17:01:17.848358 http.go:147: Attacking https://iplogger.org/*******
2022/04/05 17:01:17.865089 http.go:147: Attacking https://iplogger.org/*******
2022-04-05T17:01:18.409Z        DEBUG   job/http.go:161 error sending request   {"error": "the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection", "args": {"client":{"static_host":{"addr":"148.251.234.83:443","is_tls":"true"}},"request":{"method":"GET","path":"https://iplogger.org/2Mzjp5"}}}
2022/04/05 17:01:30.203824 main.go:137: Terminating

PerchunPak commented 2 years ago

Also, how about add timeout for switch proxies? Should I open new feature request?

arriven commented 2 years ago

proxies used in countrycheck are independent from proxies used in actual jobs, you can check that same example with iplogger (great resource btw, thanks for it!) and see that you get a bunch of different IPs there. I'd only recommend you to test it out with at least 20-50 jobs to see the effects faster

arriven commented 2 years ago

example from my testing

PerchunPak commented 2 years ago

Working, thanks

arriven / db1000n