search

constverum / ProxyBroker

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:
http://proxybroker.readthedocs.io
Apache License 2.0
3.83k stars 1.08k forks source link

Add missing features from proxybroker2 (minimum proxies in queue, remove proxy, sample script, strategy option) #161

Open bluet opened 4 years ago

bluet commented 4 years ago

From https://github.com/bluet/proxybroker2

added: --min_queue to keep a minimum number of proxy candidates (and RR before promote them to official pool) added: remove proxy by calling special url http://proxyremove/host:port added: --strategy=best command line option added: use_existing_proxy.py in example added: inject X-Proxy-Info header in response added: HTTP API for getting proxy info and removing proxy from queue added: docker image on docker hub

Related issues: https://github.com/constverum/ProxyBroker/issues/147 https://github.com/constverum/ProxyBroker/issues/142 https://github.com/constverum/ProxyBroker/issues/139

harryqt commented 4 years ago 
harry@ubuntu16:~$ pip3 install -U git+https://github.com/bluet/ProxyBroker.git
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/bluet/ProxyBroker.git
  Cloning https://github.com/bluet/ProxyBroker.git to /tmp/pip-req-build-6uybb137
  Running command git clone -q https://github.com/bluet/ProxyBroker.git /tmp/pip-req-build-6uybb137
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
ERROR: Package 'proxybroker' requires a different Python: 3.5.2 not in '>=3.5.3'
bluet commented 4 years ago

@Dibbyo456 Could you try a clean install?

It works in my test. Not sure why it failed installing at your side.

(env) bluet@ocisly:~/workspace/ProxyBroker/tmp/testmodule$ python --version
Python 3.6.9
(env) bluet@ocisly:~/workspace/ProxyBroker/tmp/testmodule$ pip3 install -U git+https://github.com/bluet/ProxyBroker.git
Collecting git+https://github.com/bluet/ProxyBroker.git
  Cloning https://github.com/bluet/ProxyBroker.git to /tmp/pip-req-build-v2286u55
  Running command git clone -q https://github.com/bluet/ProxyBroker.git /tmp/pip-req-build-v2286u55
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Processing /home/bluet/.cache/pip/wheels/e8/74/a5/1f91545399ab5055993722d7b368372c36137b67a4418e3d53/maxminddb-1.5.2-py3-none-any.whl
Collecting aiodns>=2.0.0
  Using cached aiodns-2.0.0-py2.py3-none-any.whl (4.8 kB)
Collecting aiohttp>=3.5.4
  Using cached aiohttp-3.6.2-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB)
Collecting attrs==19.1.0
  Using cached attrs-19.1.0-py2.py3-none-any.whl (35 kB)
Collecting typing; python_version < "3.7"
  Using cached typing-3.7.4.1-py3-none-any.whl (25 kB)
Collecting pycares>=3.0.0
  Using cached pycares-3.1.1-cp36-cp36m-manylinux2010_x86_64.whl (228 kB)
Collecting multidict<5.0,>=4.5
  Using cached multidict-4.7.5-cp36-cp36m-manylinux1_x86_64.whl (148 kB)
Collecting yarl<2.0,>=1.0
  Using cached yarl-1.4.2-cp36-cp36m-manylinux1_x86_64.whl (252 kB)
Processing /home/bluet/.cache/pip/wheels/6a/f5/9c/f8331a854f7a8739cf0e74c13854e4dd7b1af11b04fe1dde13/idna_ssl-1.1.0-py3-none-any.whl
Collecting typing-extensions>=3.6.5; python_version < "3.7"
  Using cached typing_extensions-3.7.4.2-py3-none-any.whl (22 kB)
Collecting chardet<4.0,>=2.0
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting async-timeout<4.0,>=3.0
  Using cached async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting cffi>=1.5.0
  Using cached cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (399 kB)
Collecting idna>=2.0
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: proxybroker
  Building wheel for proxybroker (PEP 517) ... done
  Created wheel for proxybroker: filename=proxybroker-0.3.2-py3-none-any.whl size=1570667 sha256=fbc69e4bdb9f65cb2655eb43356ac0a1fa2db62d2f24498f822423d51edad51d
  Stored in directory: /tmp/pip-ephem-wheel-cache-hj9tx0ii/wheels/61/d8/f0/3d3b59aa216f90c239a5c8ea9e9033e795fad4c1e532d2b861
Successfully built proxybroker
Installing collected packages: maxminddb, typing, pycparser, cffi, pycares, aiodns, multidict, idna, yarl, idna-ssl, typing-extensions, chardet, async-timeout, attrs, aiohttp, proxybroker
Successfully installed aiodns-2.0.0 aiohttp-3.6.2 async-timeout-3.0.1 attrs-19.1.0 cffi-1.14.0 chardet-3.0.4 idna-2.9 idna-ssl-1.1.0 maxminddb-1.5.2 multidict-4.7.5 proxybroker-0.3.2 pycares-3.1.1 pycparser-2.20 typing-3.7.4.1 typing-extensions-3.7.4.2 yarl-1.4.2
(env) bluet@ocisly:~/workspace/ProxyBroker/tmp/testmodule$ pip3 --version
pip 20.1 from /home/bluet/workspace/ProxyBroker/tmp/testmodule/env/lib/python3.6/site-packages/pip (python 3.6)