Empty Proxy List

[cbjrobertson]

I am running http_request_randomizer via your python api on python 3.6 on OSX 10.13.1. Running the method call recommended here results in the following log and error. The list of proxies is apparently empty due to various errors in the proxy list parser requests. Is this something I'm doing wrong or a bug?

2018-02-11 19:38:45,974 root   DEBUG    === Initialized Proxy Parsers ===
2018-02-11 19:38:45,977 root   DEBUG             FreeProxy Parser of 'http://free-proxy-list.net' with required bandwidth: '150' KBs
2018-02-11 19:38:45,979 root   DEBUG             ProxyForEU Parser of 'http://proxyfor.eu/geo.php' with required bandwidth: '1.0' KBs
2018-02-11 19:38:45,985 root   DEBUG             RebroWeebly Parser of 'http://rebro.weebly.com' with required bandwidth: '150' KBs
2018-02-11 19:38:45,991 root   DEBUG             SemairProxy Parser of 'https://premproxy.com/list/' with required bandwidth: '150' KBs
2018-02-11 19:38:45,994 root   DEBUG    =================================
/anaconda3/lib/python3.6/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for free-proxy-list.net has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
2018-02-11 19:38:46,140 urllib3.connection ERROR    Certificate did not match expected hostname: free-proxy-list.net. Certificate: {'subject': ((('commonName', 'sni108365.cloudflaressl.com'),),), 'subjectAltName': []}
2018-02-11 19:38:46,143 http_request_randomizer.requests.parsers.FreeProxyParser ERROR    Provider FreeProxy failed with Unknown error: HTTPSConnectionPool(host='free-proxy-list.net', port=443): Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname 'free-proxy-list.net' doesn't match 'sni108365.cloudflaressl.com'",),))
2018-02-11 19:38:46,222 http_request_randomizer.requests.parsers.ProxyForEuParser ERROR    Provider ProxyForEU failed with Attribute error: 'NoneType' object has no attribute 'find'
2018-02-11 19:38:46,590 http_request_randomizer.requests.parsers.RebroWeeblyParser WARNING  Proxy Provider url failed: http://rebro.weebly.com
/anaconda3/lib/python3.6/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for premproxy.com has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
2018-02-11 19:38:46,741 http_request_randomizer.requests.parsers.SamairProxyParser DEBUG    Pages: set()
Traceback (most recent call last):

  File "<ipython-input-35-fafe89d53dac>", line 1, in <module>
    runfile('/Users/cole/Desktop/webScraping/requestProxy.py', wdir='/Users/cole/Desktop/webScraping')

  File "/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 705, in runfile
    execfile(filename, namespace)

  File "/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/Users/cole/Desktop/webScraping/requestProxy.py", line 18, in <module>
    req_proxy = RequestProxy()

  File "/anaconda3/lib/python3.6/site-packages/http_request_randomizer/requests/proxy/requestProxy.py", line 60, in __init__
    self.current_proxy = self.randomize_proxy()

  File "/anaconda3/lib/python3.6/site-packages/http_request_randomizer/requests/proxy/requestProxy.py", line 77, in randomize_proxy
    raise ProxyListException("list is empty")

ProxyListException: list is empty

[pgaref]

Hello @cbjrobertson ,

I was unable to reproduce the error with Python 3.6.4 but I am quite sure the error is related to your python distribution (Conda) and more precisly with the way they verify SSL certificates. Source: https://github.com/conda/conda/issues/5170

I would either try their solution or a clean (non-Conda) python interpreter.

[cbjrobertson]

Solved by running in IDLE. Thank you.

[jozelazarevski]

How do you run it in IDLE? I face the same problem

[hasansezertasan]

I tried with idle and with python 3.6.4(conda), what should I do?