Some onion URL's don't seem to work

[shimmy568]

For some reason some onion URL's seem to work and others don't.

When I do it works fine:

do_request('http://nzxj65x32vh2fkhk.onion')

However this crashes the program:

do_request('http://propub3r6espa33w.onion')

The program output:

Sending: GET http://propub3r6espa33w.onion
NewConnectionError
Traceback (most recent call last):
  File "/home/darkdive/.local/lib/python3.8/site-packages/torpy/http/adapter.py", line 157, in _new_conn
    self._tor_stream = self._circuit.create_stream((self.host, self.port))
  File "/home/darkdive/.local/lib/python3.8/site-packages/torpy/circuit.py", line 320, in wrapped
    return fn(self, *args, **kwargs)
  File "/home/darkdive/.local/lib/python3.8/site-packages/torpy/circuit.py", line 599, in create_stream
    tor_stream.connect(address)
  File "/home/darkdive/.local/lib/python3.8/site-packages/torpy/stream.py", line 265, in connect
    hidden_service, address = self._prepare_address(address)
  File "/home/darkdive/.local/lib/python3.8/site-packages/torpy/stream.py", line 256, in _prepare_address
    return HiddenService(address[0], descriptor_cookie, auth_type), address
  File "/home/darkdive/.local/lib/python3.8/site-packages/torpy/hiddenservice.py", line 59, in __init__
    self._onion_address, self._permanent_id, self._onion_identity_pk = self._parse_onion(onion_address)
TypeError: cannot unpack non-iterable NoneType object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/home/darkdive/.local/lib/python3.8/site-packages/torpy/http/adapter.py", line 167, in _new_conn
    raise NewConnectionError(self, 'Failed to establish a new connection: %s' % e)
urllib3.exceptions.NewConnectionError: <torpy.http.adapter.MyHTTPSConnection object at 0x7fccb0ca13d0>: Failed to establish a new connection: cannot unpack non-iterable NoneType object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: MyHTTPSConnectionPool(host='www.propub3r6espa33w.onion', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<torpy.http.adapter.MyHTTPSConnection object at 0x7fccb0ca13d0>: Failed to establish a new connection: cannot unpack non-iterable NoneType object'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 47, in <module>
    do_request(hostname2)
  File "/home/darkdive/.local/lib/python3.8/site-packages/torpy/http/requests.py", line 80, in do_request
    response = s.send(request.prepare())
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 668, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 239, in resolve_redirects
    resp = self.send(
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: MyHTTPSConnectionPool(host='www.propub3r6espa33w.onion', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<torpy.http.adapter.MyHTTPSConnection object at 0x7fccb0ca13d0>: Failed to establish a new connection: cannot unpack non-iterable NoneType object'))

It seems like this function is getting a url format that it doesn't expect and returns None

Any input would be greatly appreciated

[jbrown299]

Yep, there is bug inside onion hostname normalization.

But anyway your address will not work because it's redirecting to v3 hidden service which not supported for now:

Sending: GET http://propub3r6espa33w.onion

send: b'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: propub3r6espa33w.onion\r\nConnection: close\r\n\r\n'
...
reply: 'HTTP/1.1 301 Moved Permanently\r\n'
header: Server: nginx
header: Content-Type: text/html
header: Content-Length: 178
header: Connection: close
header: Location: https://www.propub3r6espa33w.onion/
...
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
header: Server: nginx/1.14.0 (Ubuntu)
header: Content-Type: text/html
header: Content-Length: 170
header: Connection: close
header: Location: https://p53lf57qovyuvwsc6xnrppyply3vtqm7l6pcobkmyqsiofyeznfu5uqd.onion/
...
Exception: v3 onion hidden service not supported yet