Lookyloo / lookyloo

Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.
https://www.lookyloo.eu
Other
664 stars 83 forks source link

[Bug]: push to misp silently fails accessing VT behind a proxy #896

Closed DocArmoryTech closed 3 months ago

DocArmoryTech commented 3 months ago

What happened?

Environment: a private domain with no default route, a LACUS instance, and an http proxy configured using environment variables (HTTP_PROXY, HTTPS_PROXY, NO_PROXY)

Using the web-ui and "preparing a push to MISP" the modal dialog appears, fails to expand, and appears to hang. Logs (/website/logs/warning.log) reveal the VT module is unable to reach www.virustotal.com, i.e. the module is not using the environmentally configured http_proxy.

vt-py docs for the Client indicate the trustenv parameter is False by default i.e. the VT Client does not use/trust proxy environment variables.

If the issue is related to the web interface, what browsers are you seeing the problem on?

No response

Relevant log output

Traceback (most recent call last):
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/usr/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
TimeoutError: [Errno 110] Connect call failed ('74.125.34.46', 443)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/flask_restx/api.py", line 674, in error_router
    return original_handler(e)
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/flask_cors/extension.py", line 176, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/flask_login/utils.py", line 290, in decorated_view
    return current_app.ensure_sync(func)(*args, **kwargs)
  File "/home/looky/lookyloo/website/web/__init__.py", line 376, in web_misp_push_view
    event = lookyloo.misp_export(tree_uuid)
  File "/home/looky/lookyloo/lookyloo/lookyloo.py", line 1155, in misp_export
    vt_obj = self.__misp_add_vt_to_URLObject(e_obj)
  File "/home/looky/lookyloo/lookyloo/lookyloo.py", line 1104, in __misp_add_vt_to_URLObject
    self.vt.url_lookup(url.value)
  File "/home/looky/lookyloo/lookyloo/modules/vt.py", line 93, in url_lookup
    url_information = self.client.get_object(f"/urls/{url_id}")
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/vt/client.py", line 534, in get_object
    return make_sync(self.get_object_async(path, *path_args, params=params))
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/vt/utils.py", line 27, in make_sync
    return event_loop.run_until_complete(future)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/vt/client.py", line 538, in get_object_async
    response = await self.get_async(path, *path_args, params=params)
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/vt/client.py", line 440, in get_async
    await self._get_session().get(
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/aiohttp/client.py", line 562, in _request
    conn = await self._connector.connect(
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/aiohttp/connector.py", line 1209, in _create_direct_connection
    raise last_exc
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/aiohttp/connector.py", line 1178, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/home/looky/.cache/pypoetry/virtualenvs/lookyloo-Ai6b8dBN-py3.8/lib/python3.8/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.virustotal.com:443 ssl:default [Connect call failed ('74.125.34.46', 443)]
Rafiot commented 3 months ago

Good catch, thanks!