search

blacklanternsecurity / bbot

A recursive internet scanner for hackers.
https://www.blacklanternsecurity.com/bbot/
GNU General Public License v3.0
6.43k stars 501 forks source link

tldsextract library causing SSL verify errors in MITM environment #1

Closed liquidsec closed 2 years ago

liquidsec commented 2 years ago

The tldsextract library is causing SSL verify errors even with sslverify=False in config when in an ssl-proxied (mitm) environment As shown in the stack trace when operating in an environment where all traffic is being intercepted this module makes a call via requests to https://publicsuffix.org/list/public_suffix_list.dat and https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat. These calls do not respect the sslverify=False setting which the rest of bbot respects (such as web request helper).

At the end of the stack trace, --current-config was run to show that ssl_verify is currently false.

Issue appears to be rooted in the tldsextract library.

root@c0965812342f:/opt/bbot# poetry run bbot -d -m httpx -t https://icanhazip.com 
[INFO] bbot.cli: Command: bbot -d -m httpx -t https://icanhazip.com
[DBUG] bbot.core.event: Autodetected event type "URL" based on data: "https://icanhazip.com"
[INFO] bbot.scanner: Loading 1 modules: httpx
[INFO] bbot.scanner: Loaded module "httpx"
[SUCC] bbot.scanner: Loaded 1 modules
[INFO] bbot.scanner: Starting scan asdf
[INFO] bbot.scanner: Setting up modules
[DBUG] bbot.modules.httpx: Setting up module httpx
[DBUG] bbot.modules.httpx: Finished setting up module httpx
[INFO] bbot.scanner: Finished setting up modules
[INFO] bbot.scanner: Target: Event("URL", "https://icanhazip.com")
{"type": "URL", "data": "https://icanhazip.com", "module": "module", "source": "f762108ca727f0bcd961fc7467c248887dfa70dc:TARGET", "id": "baed85c983e6564d59e4ca4cbf62df4d78d3492d:URL", "tags": ["target"]}
[INFO] bbot.scanner: Starting modules
[INFO] bbot.scanner: 1 modules started
Exception reading Public Suffix List url https://publicsuffix.org/list/public_suffix_list.dat
Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 190, in run_and_cache
    result = self.get(namespace=namespace, key=key_args)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 93, in get
    raise KeyError("namespace: " + namespace + " key: " + repr(key))
KeyError: "namespace: publicsuffix.org-tlds key: {'urls': ('https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 190, in run_and_cache
    result = self.get(namespace=namespace, key=key_args)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 93, in get
    raise KeyError("namespace: " + namespace + " key: " + repr(key))
KeyError: "namespace: urls key: {'url': 'https://publicsuffix.org/list/public_suffix_list.dat'}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='publicsuffix.org', port=443): Max retries exceeded with url: /list/public_suffix_list.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/suffix_list.py", line 30, in find_first_response
    return cache.cached_fetch_url(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 199, in cached_fetch_url
    return self.run_and_cache(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 192, in run_and_cache
    result = func(**kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 209, in _fetch_url
    response = session.get(url, timeout=timeout)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='publicsuffix.org', port=443): Max retries exceeded with url: /list/public_suffix_list.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
Exception reading Public Suffix List url https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat
Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 190, in run_and_cache
    result = self.get(namespace=namespace, key=key_args)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 93, in get
    raise KeyError("namespace: " + namespace + " key: " + repr(key))
KeyError: "namespace: publicsuffix.org-tlds key: {'urls': ('https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 190, in run_and_cache
    result = self.get(namespace=namespace, key=key_args)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 93, in get
    raise KeyError("namespace: " + namespace + " key: " + repr(key))
KeyError: "namespace: urls key: {'url': 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /publicsuffix/list/master/public_suffix_list.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/suffix_list.py", line 30, in find_first_response
    return cache.cached_fetch_url(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 199, in cached_fetch_url
    return self.run_and_cache(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 192, in run_and_cache
    result = func(**kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 209, in _fetch_url
    response = session.get(url, timeout=timeout)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /publicsuffix/list/master/public_suffix_list.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
[VERB] bbot.scanner: Events queued: 0 (None)
[VERB] bbot.scanner: Tasks queued: 0 (None)
[DBUG] bbot.modules.httpx: Handling batch of 1 events for module "httpx"
[SUCC] bbot.scanner: Scan asdf completed with status FINISHED
root@c0965812342f:/opt/bbot# poetry run bbot --current-config
modules:
  httpx:
    allow_skip_portscan: true
  nuclei: {}
  sublist3r: {}
  sslcert:
    timeout: 4.0
  dnsresolve:
    max_hosts: 65536
  naabu: {}
  aspnet_viewstate: {}
  dnsx:
    wordlist: https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/DNS/subdomains-top1million-20000.txt
  wayback: {}
  dnsdumpster: {}
max_threads: 250
http_proxy: null
http_timeout: 30
ssl_verify: false
user_agent: Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15
  (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1
dns_wildcard_tests: 5
url_extension_blacklist:
- png
- jpg
- jpeg
- gif
- svg
- css
- woff
- woff2
- ttf

root@c0965812342f:/opt/bbot#
TheTechromancer commented 2 years ago

Fixed in cff6a7bb50b6e777ecc281d586af5038d4fa8e3e