search

bluet / proxybroker2

The New (auto rotate) Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:
Apache License 2.0
708 stars 112 forks source link

`Serve` Does not work. #153

Open sequencerr opened 8 months ago

sequencerr commented 8 months ago 
@debian:~$ docker run --rm -p 8888:8888 bluet/proxybroker2 --log DEBUG sargument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.erve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High --min-queue 
5
/usr/local/lib/python3.9/asyncio/events.py:80: DeprecationWarning: The loop og CRITICAL serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High --min-qu
argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.                                                                      erve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High --min-queue 5        
  self._context.run(self._callback, *self._args)                            argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
Server started at http://127.0.0.1:8888
[11:42:29] - INFO - proxybroker - Listening established on ('127.0.0.1', 8888)                                                                          8)
@debian:~$ curl http://api.ipify.org -x http://127.0.0.1:8888
curl: (56) Recv failure: Connection reset by peer
@debian:~$ curl http://127.0.0.1:8888
curl: (56) Recv failure: Connection reset by peer
@debian:~$

Screenshots If applicable, add screenshots to help explain your problem. image

Desktop (please complete the following information):

  Host: debian Kernel: 6.1.0-12-amd64 arch: x86_64 bits: 64
    Console: pty pts/1 Distro: Debian GNU/Linux 12 (bookworm)
Machine:
  Type: Vmware System: VMware product: VMware Virtual Platform v: N/A       
    serial: VMware-56 4d 40 b2 aa 1c 86-43 d9 b6 ea 8f e3 b0 45
  Mobo: Intel model: 440BX Desktop Reference Platform serial: N/A
    BIOS: Phoenix v: 6.00 date: 11/12/2020
sequencerr commented 8 months ago

Well, it seems nothing works for me now

yrch@debian:~$ docker rmi bluet/proxybroker2 -f
Untagged: bluet/proxybroker2:latest
Untagged: bluet/proxybroker2@sha256:fa6c5f1872e27e55c1a9a83d2a651746b51a13ccd756c7744bdd2019ab150a9e
Deleted: sha256:b992776849e7182b6e12168d28feff072fced901c92a232d26a826ed99bd382b
yrch@debian:~$ docker run -p 8081:8081 -v --rm bluet/proxybroker2 find --types HTTP --lvl {Anonymous,High} --strict -f txt
Unable to find image 'bluet/proxybroker2:latest' locally
latest: Pulling from bluet/proxybroker2
1f7ce2fa46ab: Already exists
442c5d63eafd: Already exists
342f11d60b1e: Already exists
57b3e82cff54: Already exists
ec926eb426a4: Already exists
b78bea8ce2ab: Already exists
2d5aab209527: Already exists
23640612747f: Already exists
03a99065a2a7: Already exists
33b4d4d22b9e: Already exists
Digest: sha256:fa6c5f1872e27e55c1a9a83d2a651746b51a13ccd756c7744bdd2019ab150a9e
Status: Downloaded newer image for bluet/proxybroker2:latest
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/app/proxybroker/__main__.py", line 4, in <module>
    cli()
  File "/app/proxybroker/cli.py", line 446, in cli
    loop.run_until_complete(asyncio.gather(*tasks))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/app/proxybroker/api.py", line 176, in find
    ip = await self._resolver.get_real_ext_ip()
  File "/app/proxybroker/resolver.py", line 118, in get_real_ext_ip
    raise RuntimeError('Could not get the external IP')
RuntimeError: Could not get the external IP

geo

yrch@debian:~$ proxybroker update-geo
The update in progress, please waite for a while...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.9/socket.py", line 823, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/app/proxybroker/__main__.py", line 4, in <module>
    cli()
  File "/app/proxybroker/cli.py", line 377, in cli
    ns.func()
  File "/app/proxybroker/utils.py", line 120, in update_geoip_db
    urllib.request.urlretrieve(url, local_file)
  File "/usr/local/lib/python3.9/urllib/request.py", line 239, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/local/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/local/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.9/urllib/request.py", line 1375, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/local/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
yrch@debian:~$ proxybroker update-geo --log DEBUG
usage: proxybroker [--max-conn MAX_CONN] [--max-tries MAX_TRIES]
                   [--timeout SECONDS] [--judge JUDGES] [--provider PROVIDERS]
                   [--verify-ssl]
                   [--log [{NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]]
                   [--version] [--help]
                   {find,grab,serve,update-geo} ...
proxybroker: error: unrecognized arguments: --log DEBUG
$ docker pull bluet/proxybroker2
Using default tag: latest
latest: Pulling from bluet/proxybroker2
Digest: sha256:fa6c5f1872e27e55c1a9a83d2a651746b51a13ccd756c7744bdd2019ab150a9e
Status: Image is up to date for bluet/proxybroker2:latest
docker.io/bluet/proxybroker2:latestyrch@debian:~$ proxybroker update-geo
The update in progress, please waite for a while...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.9/socket.py", line 823, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/app/proxybroker/__main__.py", line 4, in <module>
    cli()
  File "/app/proxybroker/cli.py", line 377, in cli
    ns.func()
  File "/app/proxybroker/utils.py", line 120, in update_geoip_db
    urllib.request.urlretrieve(url, local_file)
  File "/usr/local/lib/python3.9/urllib/request.py", line 239, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/local/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/local/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.9/urllib/request.py", line 1375, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/local/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
yrch@debian:~$ proxybroker update-geo --log DEBUG
usage: proxybroker [--max-conn MAX_CONN] [--max-tries MAX_TRIES]
                   [--timeout SECONDS] [--judge JUDGES] [--provider PROVIDERS]
                   [--verify-ssl]
                   [--log [{NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]]
                   [--version] [--help]
                   {find,grab,serve,update-geo} ...
proxybroker: error: unrecognized arguments: --log DEBUG
$ docker pull bluet/proxybroker2
Using default tag: latest
latest: Pulling from bluet/proxybroker2
Digest: sha256:fa6c5f1872e27e55c1a9a83d2a651746b51a13ccd756c7744bdd2019ab150a9e
Status: Image is up to date for bluet/proxybroker2:latest
docker.io/bluet/proxybroker2:latest
ziloka commented 7 months ago

The temporary failure of name resolution.. I believe something is end on your end.

The --log DEBUG argument is suppose to be before the subcommand

eg proxybroker --log DEBUG update-geo

This seems related with #147

amiremami commented 7 months ago

Hi, I got this error:

proxybroker serve --host 127.0.0.1 --port 8888 -ssl -t 1 --max-conn 10 --max-tries 1 --types HTTP HTTPS -l 7000

usage: proxybroker serve [--host HOST] [--port PORT] [--max-tries SRV_MAX_TRIES] [--strategy STRATEGY] [--min-queue MIN_QUEUE] [--min-req-proxy MIN_REQ_PROXY] [--max-error-rate MAX_ERROR_RATE] [--max-resp-time SECONDS] [--prefer-connect] [--http-allowed-codes HTTP_ALLOWED_CODES [HTTP_ALLOWED_CODES ...]] [--backlog BACKLOG] --types {HTTP,HTTPS,SOCKS4,SOCKS5,CONNECT:80,CONNECT:25} [{HTTP,HTTPS,SOCKS4,SOCKS5,CONNECT:80,CONNECT:25} ...] [--lvl {Transparent,Anonymous,High} [{Transparent,Anonymous,High} ...]] [--data DATA] [--dnsbl DNSBL [DNSBL ...]] [--post] [--strict] [--countries COUNTRIES [COUNTRIES ...]] [--limit LIMIT] [--help] proxybroker serve: error: argument --limit/-l: expected one argument root@ubuntu:~# proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High --min-queue 5 Traceback (most recent call last): File "/usr/local/bin/proxybroker", line 8, in sys.exit(cli()) File "/usr/local/lib/python3.10/dist-packages/proxybroker/cli.py", line 423, in cli broker.serve( File "/usr/local/lib/python3.10/dist-packages/proxybroker/api.py", line 295, in serve self._server.start() File "/usr/local/lib/python3.10/dist-packages/proxybroker/server.py", line 157, in start self._server = self._loop.run_until_complete(srv) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/usr/lib/python3.10/asyncio/streams.py", line 85, in start_server return await loop.create_server(factory, host, port, **kwds) TypeError: BaseEventLoop.create_server() got an unexpected keyword argument 'loop'

techux commented 6 months ago

Hi, I got this error:

proxybroker serve --host 127.0.0.1 --port 8888 -ssl -t 1 --max-conn 10 --max-tries 1 --types HTTP HTTPS -l 7000

usage: proxybroker serve [--host HOST] [--port PORT] [--max-tries SRV_MAX_TRIES] [--strategy STRATEGY] [--min-queue MIN_QUEUE] [--min-req-proxy MIN_REQ_PROXY] [--max-error-rate MAX_ERROR_RATE] [--max-resp-time SECONDS] [--prefer-connect] [--http-allowed-codes HTTP_ALLOWED_CODES [HTTP_ALLOWED_CODES ...]] [--backlog BACKLOG] --types {HTTP,HTTPS,SOCKS4,SOCKS5,CONNECT:80,CONNECT:25} [{HTTP,HTTPS,SOCKS4,SOCKS5,CONNECT:80,CONNECT:25} ...] [--lvl {Transparent,Anonymous,High} [{Transparent,Anonymous,High} ...]] [--data DATA] [--dnsbl DNSBL [DNSBL ...]] [--post] [--strict] [--countries COUNTRIES [COUNTRIES ...]] [--limit LIMIT] [--help] proxybroker serve: error: argument --limit/-l: expected one argument root@ubuntu:~# proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High --min-queue 5 Traceback (most recent call last): File "/usr/local/bin/proxybroker", line 8, in sys.exit(cli()) File "/usr/local/lib/python3.10/dist-packages/proxybroker/cli.py", line 423, in cli broker.serve( File "/usr/local/lib/python3.10/dist-packages/proxybroker/api.py", line 295, in serve self._server.start() File "/usr/local/lib/python3.10/dist-packages/proxybroker/server.py", line 157, in start self._server = self._loop.run_until_complete(srv) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/usr/lib/python3.10/asyncio/streams.py", line 85, in start_server return await loop.create_server(factory, host, port, **kwds) TypeError: BaseEventLoop.create_server() got an unexpected keyword argument 'loop'

I also get same error

Kolliderrr commented 6 months ago

@TechUX ,

I am novice at python, so it's open to discuss.

As i marked in exception, main problem for me ws same BaseEventLoop:

└─Δ proxybroker2 serve --host 127.0.0.1 --port 8889 --types HTTP HTTPS
Traceback (most recent call last):
  File "/home/valievar/.local/bin/proxybroker2", line 8, in <module>
    sys.exit(cli())
  File "/home/user/.local/lib/python3.10/site-packages/proxybroker2/cli.py", line 420, in cli
    broker.serve(
  File "/home/user/.local/lib/python3.10/site-packages/proxybroker2/api.py", line 295, in serve
    self._server.start()
  File "/home/user/.local/lib/python3.10/site-packages/proxybroker2/server.py", line 157, in start
    self._server = self._loop.run_until_complete(srv)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/lib/python3.10/asyncio/streams.py", line 85, in start_server
    return await loop.create_server(factory, host, port, **kwds)
TypeError: BaseEventLoop.create_server() got an unexpected keyword argument 'loop'

Being curious of that, i changed the part in.local/lib/python3.10/site-packages/proxybroker2/server.py" in 148 row: class Server:

    """Server distributes incoming requests to a pool of found proxies."""

    def __init__(
        self,
        host,
        port,
        proxies,
        timeout=8,
        max_tries=3,
        min_queue=5,
        min_req_proxy=5,
        max_error_rate=0.5,
        max_resp_time=8,
        prefer_connect=False,
        http_allowed_codes=None,
        backlog=100,
        loop=None,
        **kwargs,
    ):
        self.host = host
        self.port = int(port)
        self._loop = loop or asyncio.get_event_loop()
        self._timeout = timeout
        self._max_tries = max_tries
        self._backlog = backlog
        self._prefer_connect = prefer_connect

        self._server = None
        self._connections = {}
        self._proxy_pool = ProxyPool(
            proxies, min_req_proxy, max_error_rate, max_resp_time, min_queue
        )
        self._resolver = Resolver(loop=self._loop)
        self._http_allowed_codes = http_allowed_codes or []

    def start(self):

        srv = asyncio.start_server(
            self._accept,
            host=self.host,
            port=self.port,
            backlog=self._backlog
            # loop=self._loop,
        )
        self._server = self._loop.run_until_complete(srv)

        log.info(
            'Listening established on {0}'.format(self._server.sockets[0].getsockname())
        )

Just commented row 'loop' in asyncio.start_server() and it worked somehow:

proxybroker2 serve --host 127.0.0.1 --port 8889 --types HTTP HTTPS
Server started at http://127.0.0.1:8889
/usr/lib/python3.10/asyncio/trsock.py:20: DeprecationWarning: Using send() method on sockets returned from get_extra_info('socket') will be prohibited in asyncio 3.9. Please report your use case to bugs.python.org.
  warnings.warn(
DeprecationWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/asyncio/trsock.py:20: DeprecationWarning: Using recv() method on sockets returned from get_extra_info('socket') will be prohibited in asyncio 3.9. Please report your use case to bugs.python.org.
  warnings.warn(
DeprecationWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/asyncio/trsock.py:20: DeprecationWarning: Using close() method on sockets returned from get_extra_info('socket') will be prohibited in asyncio 3.9. Please report your use case to bugs.python.org.
  warnings.warn(
DeprecationWarning: Enable tracemalloc to get the object allocation traceback

The question is now: is it working? I'll return with update

UPD: Also, WSL2 and python version:

└─Δ python3 --version
Python 3.10.12

Ubuntu 22.04.3 LTS

UPD2: From asyncio documentation:

coroutine asyncio.start_server(client_connected_cb, host=None, port=None, *, limit=None, family=socket.AF_UNSPEC, flags=socket.AI_PASSIVE, sock=None, backlog=100, ssl=None, reuse_address=None, reuse_port=None, ssl_handshake_timeout=None, ssl_shutdown_timeout=None, start_serving=True)
Start a socket server.

The client_connected_cb callback is called whenever a new client connection is established. It receives a (reader, writer) pair as two arguments, instances of the [StreamReader](https://docs.python.org/3/library/asyncio-stream.html#asyncio.StreamReader) and [StreamWriter](https://docs.python.org/3/library/asyncio-stream.html#asyncio.StreamWriter) classes.

client_connected_cb can be a plain callable or a [coroutine function](https://docs.python.org/3/library/asyncio-task.html#coroutine); if it is a coroutine function, it will be automatically scheduled as a [Task](https://docs.python.org/3/library/asyncio-task.html#asyncio.Task).

limit determines the buffer size limit used by the returned [StreamReader](https://docs.python.org/3/library/asyncio-stream.html#asyncio.StreamReader) instance. By default the limit is set to 64 KiB.

The rest of the arguments are passed directly to [loop.create_server()](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.create_server).

Note The sock argument transfers ownership of the socket to the server created. To close the socket, call the server’s [close()](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Server.close) method.
Changed in version 3.7: Added the ssl_handshake_timeout and start_serving parameters.

Changed in version 3.10: Removed the loop parameter.

Changed in version 3.11: Added the ssl_shutdown_timeout parameter.

So, from the version 3.10, now asyncio.start_server use current loop or smthing

CatWithAutism commented 4 months ago

You have to bind a local server in your container to 0.0.0.0

Here is a working example:

docker run -p 0.0.0.0:8888:8888 -it bluet/proxybroker2 --log INFO --max-conn 50 --max-tries 2 --timeout 5 serve --host 0.0.0.0 --port 8888 --types HTTPS --lvl High --min-queue 10

Kolliderrr commented 4 months ago

You have to bind a local server in your container to 0.0.0.0

Here is a working example:

docker run -p 0.0.0.0:8888:8888 -it bluet/proxybroker2 --log INFO --max-conn 50 --max-tries 2 --timeout 5 serve --host 0.0.0.0 --port 8888 --types HTTPS --lvl High --min-queue 10

I'll try that, that's real appreciated. Tnx

jkroepke commented 4 months ago

Doesnt work for me.

Can the image also be available as arm64?

docker run -p 0.0.0.0:8888:8888 -it bluet/proxybroker2 --log INFO --max-conn 50 --max-tries 2 --timeout 5 serve --host 0.0.0.0 --port 8888 --types HTTPS --lvl High --min-queue 10
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /usr/local/bin/python: exec format error
CatWithAutism commented 4 months ago

Doesnt work for me.

Can the image also be available as arm64?

docker run -p 0.0.0.0:8888:8888 -it bluet/proxybroker2 --log INFO --max-conn 50 --max-tries 2 --timeout 5 serve --host 0.0.0.0 --port 8888 --types HTTPS --lvl High --min-queue 10
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /usr/local/bin/python: exec format error

I use it on arm64 with the latest version of macos and the latest version proxybroker's image

image
CatWithAutism commented 4 months ago

Doesnt work for me.

Can the image also be available as arm64?

docker run -p 0.0.0.0:8888:8888 -it bluet/proxybroker2 --log INFO --max-conn 50 --max-tries 2 --timeout 5 serve --host 0.0.0.0 --port 8888 --types HTTPS --lvl High --min-queue 10
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /usr/local/bin/python: exec format error

By the way you can try to use previous version It supports arm64

docker pull bluet/proxybroker2:v2.0.0-alpha6

https://hub.docker.com/r/bluet/proxybroker2/tags