kitabisa / mubeng

An incredibly fast proxy checker & IP rotator with ease.
Apache License 2.0
1.63k stars 194 forks source link
http-proxy https-proxy ip-rate-limit ip-rotation proxy proxy-checker proxy-list proxy-rotation proxy-rotator proxy-server socks5 socks5-proxy


mubeng

An incredibly fast proxy checker & IP rotator with ease.

ContributingWhat's newDocumentationReport Issues



Features

Why mubeng?

It's fairly simple, there is no need for additional configuration.

mubeng has 2 core functionality:

1. Run proxy server as proxy IP rotation

This is useful to avoid different kinds of IP ban, i.e. bruteforce protection, API rate-limiting or WAF blocking based on IP. We also leave it entirely up to user to use proxy pool resources from anywhere.

2. Perform proxy checks

So, you don't need any extra proxy checking tools out there if you want to check your proxy pool.

Installation

Binary

Simply, download a pre-built binary from releases page and run!

Docker

Pull the Docker image by running:

▶ docker pull ghcr.io/kitabisa/mubeng:latest

Source

Using Go compiler:

▶ go install -v github.com/kitabisa/mubeng/cmd/mubeng@latest

— or

Manual building executable from source code:

▶ git clone https://github.com/kitabisa/mubeng
▶ cd mubeng
▶ make build
▶ (sudo) install ./bin/mubeng /usr/local/bin

Usage

For usage, it's always required to provide your proxy list, whether it is used to check or as a proxy pool for your proxy IP rotation.

kitabisa mubeng

Basic

▶ mubeng [-c|-a :8080] -f file.txt [options...]

Options

Here are all the options it supports.

▶ mubeng -h
Flag Description
-f, --file <FILE> Proxy file.
-a, --address <ADDR>:<PORT> Run proxy server.
-A, --auth <USER>:<PASS> Set authorization for proxy server.
-d, --daemon Daemonize proxy server.
-c, --check To perform proxy live check.
-g, --goroutine <N> Max. goroutine to use (default: 50).
--only-cc <AA>,<BB> Only show specific country code (comma separated).
-t, --timeout Max. time allowed for proxy server/check (default: 30s).
-r, --rotate <AFTER> Rotate proxy IP for every AFTER request (default: 1).
--rotate-on-error Rotate proxy IP and retry failed HTTP requests.
--remove-on-error Remove proxy IP from proxy pool on failed HTTP requests.
--max-errors <N> Max. errors allowed during rotation (default: 3).
Use this with --rotate-on-error.
If value is less than 0 (e.g., -1), rotation will
continue indefinitely.
--max-redirs <N> Max. redirects allowed (default: 10).
--max-retries <N> Max. retries for failed HTTP requests (default: 0).
-m, --method <METHOD> Rotation method (sequent/random) (default: sequent).
-s, --sync Sync will wait for the previous request to complete.
-v, --verbose Dump HTTP request/responses or show died proxy on check.
-o, --output <FILE> Save output from proxy server or live check.
-u, --update Update mubeng to the latest stable version.
-w, --watch Watch proxy file, live-reload from changes.
-V, --version Show current mubeng version.

NOTES:

  • Rotations are counted for all requests, even if the request fails.
  • The proxy server runs asynchronously by default, so it doesn't guarantee that your requests after N (which is N+1 and so on) will rotate the proxy IP, instead use the -s/--sync flag to wait for requests to the previous proxy to complete.
  • Daemon mode (-d/--daemon) will install mubeng as a service on the (Linux/OSX) system/setting up callback (Windows).
    • Hence you can control service with journalctl, service or net (for Windows) command to start/stop proxy server.
    • Whenever you activate the daemon mode, it works by forcibly stop and uninstalling the existing mubeng service, then re-install and starting it up in daemon.
  • Verbose mode (-v/--verbose) and timeout (-t/--timeout) apply to both proxy check and proxy IP rotation actions.
  • HTTP traffic requests and responses is displayed when verbose mode (-v/--verbose) is enabled, but
    • We DO NOT explicitly display the request/response body, and
    • All cookie values in headers will be redacted automatically.
  • If you use output option (-o/--output) to run proxy IP rotator, request/response headers are NOT written to the log file.
  • A timeout option (-t/--timeout) value is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "5s", "300ms", "-1.5h" or "2h45m".
    • Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", and "h".
  • The max. errors (--max-errors) ≠ max. retries (--max-retries).
    • The max. retries (--max-retries) apply to retrying a failed HTTP request with the same proxy.
    • Meanwhile, max. errors (--max-errors) refer to the total failed HTTP requests from the proxies used.
    • For example, once the max. retries (--max-retries) are reached (if supplied), the error is counted, and the proxy IP will rotate for the same failed HTTP request until the max. errors (--max-errors) are reached, whereas max. retries (--max-retries) DO NOT trigger a proxy rotation.
  • If the value of max. errors (--max-errors) is less than 0 (e.g. -1), rotation will continue indefinitely.

Install SSL Certificate

mubeng uses built-in certificate authority by GoProxy. With mubeng proxy server running, the generated certificate can be exported by visiting http://mubeng/cert in a browser.

Installation steps for CA certificate is similar to other proxy tools.

Examples

For example, you've proxy pool (proxies.txt) as:

http://127.0.0.1:8080
https://127.0.0.1:443
socks4://127.0.0.1:4145
socks5://127.0.0.1:2121
...
...

Because we use auto-switch transport, mubeng can accept multiple proxy protocol schemes at once.
Please refer to documentation for this package.

Proxy checker

Pass --check flag in command to perform proxy checks:

▶ mubeng -f proxies.txt --check --only-cc AU,US,UK --output live.txt

The above case also uses --output flag to save a live proxy of specific country code with --only-cc flag (ISO-3166 alpha-2) into file (live.txt) from checking result.


(Figure: Checking proxies mubeng with max. 5s timeout)

Proxy IP rotator

Furthermore, if you wish to do proxy IP rotator from proxies that are still alive earlier from the results of checking (live.txt) (or if you have your own list), you must use -a (--address) flag instead to run proxy server:

▶ mubeng -a localhost:8089 -f live.txt -r 10 -m random

The -r (--rotate) flag works to rotate your IP for every N request value you provide (10), and the -m (--method) flag will rotate the proxy sequential/randomly.

(Figure: Running mubeng as proxy IP rotator with verbose mode)

Burp Suite Upstream Proxy

In case you want to use mubeng (proxy IP rotator) as an upstream proxy in Burp Suite, acting in-between Burp Suite and mubeng to the internet, so you don't need any additional extensions in Burp Suite for that. To demonstrate this:

(Figure: Settings Burp Suite Upstream Proxy to mubeng)

In your Burp Suite instance, select Project options menu, and click Connections tab. In the Upstream Proxy Servers section, check Override user options then press Add button to add your upstream proxy rule. After that, fill required columns (Destination host, Proxy host & Proxy port) with correct details. Click OK to save settings.

OWASP ZAP Proxy Chain

It acts the same way when you using an upstream proxy. OWASP ZAP allows you to connect to another proxy for outgoing connections in OWASP ZAP session. To chain it with a mubeng proxy server:

(Figure: Settings proxy chain connection in OWASP ZAP to mubeng)

Select Tools in the menu bar in your ZAP session window, then select the Options (shortcut: Ctrl+Alt+O) submenu, and go to Connection section. In that window, scroll to Use proxy chain part then check Use an outgoing proxy server. After that, fill required columns (Address/Domain Name & Port) with correct details. Click OK to save settings.

Proxy format

Currently mubeng supports HTTP(S) & SOCKSv4(A)/v5 protocol, see examples above. But, not limited by that we also support proxy string substitution and helper functions for your proxy pool.

Templating

If you have an authenticated proxy, you definitely don't want to write credentials constantly to the proxy pool file. mubeng can evaluate environment variable with {{VARIABLE}} writing style.

For example:

  1. String substitute
$ export USERNAME="FOO"
$ export PASSWORD="BAR"
$ echo "http://{{USERNAME}}:{{PASSWORD}}@192.168.0.1:31337" > list.txt
$ mubeng -f list.txt -a :8080
  1. Helper function

Available functions currently supported:

Those following above functions are thread-safe pseudo-randomness.

As an example of its use, we will be utilizing stream isolation over Tor SOCKS. With this method, you just need one Tor instance and each request can use a different stream with a different exit node, but that doesn't guarantee that your ass will be rotated. Thus, we have to create unique USER:PASS pair to isolate streams for every connection. In order to pass pseudo-random proxy authorization, use uint32 or uint32n function on your proxy pool, like:

$ echo "socks5://{{uint32}}:{{uint32}}@127.0.0.1:9050" > list.txt
$ while :; do mubeng -f list.txt -c 2>/dev/null; done
[LIVE] [XX] [23.**.177.2] socks5://2123347975:3094119616@127.0.0.1:9050
[LIVE] [XX] [199.**.253.156] socks5://1646373938:2740927425@127.0.0.1:9050
[LIVE] [XX] [185.**.101.137] socks5://814036283:1382144874@127.0.0.1:9050
[LIVE] [XX] [185.**.83.83] socks5://2895805939:2276057153@127.0.0.1:9050
[LIVE] [XX] [103.**.167.10] socks5://408584795:1244204083@127.0.0.1:9050
[LIVE] [XX] [198.**.84.99] socks5://3015151335:251835794@127.0.0.1:9050
[LIVE] [XX] [179.**.159.197] socks5://3952852758:324998250@127.0.0.1:9050
^C

Limitations

Currently IP rotation runs the proxy server only as an HTTP protocol, not a SOCKSv4(A)/v5 protocol, even though the resource you have is SOCKSv4(A)/v5. In other words, the SOCKSv4(A)/v5 resource that you provide is used properly because it uses auto-switch transport on the client, but this proxy server DOES NOT switch to anything other than HTTP protocol.

Contributors

contributions

This project exists thanks to all the people who contribute. To learn how to setup a development environment and for contribution guidelines, see CONTRIBUTING.md.

Pronunciation

jv_ID/mo͞oˌbēNG/ — mubeng-mubeng nganti mumet. (ꦩꦸꦧꦺꦁ​ꦔꦤ꧀ꦠꦶ​ꦩꦸꦩꦺꦠ꧀)

Changes

For changes, see CHANGELOG.md.

License

This program is developed and maintained by members of Kitabisa Security Team, and this is not an officially supported Kitabisa product. This program is free software: you can redistribute it and/or modify it under the terms of the Apache license. Kitabisa mubeng and any contributions are copyright © by Dwi Siswanto 2021-2022.