blacklanternsecurity / bbot

A recursive internet scanner for hackers.
https://www.blacklanternsecurity.com/bbot/
GNU General Public License v3.0
4k stars 366 forks source link

Dependencies fail to install in BBOT 2.0 #1508

Closed TheTechromancer closed 4 days ago

TheTechromancer commented 4 days ago

Courtesy of @amiremami

# bbot -t bugz.zip -m httpx -d
  ______  _____   ____ _______
 |  ___ \|  __ \ / __ \__   __|
 | |___) | |__) | |  | | | |
 |  ___ <|  __ <| |  | | | |
 | |___) | |__) | |__| | | |
 |______/|_____/ \____/  |_|
 BIGHUGE BLS OSINT TOOL v2.0.0

 www.blacklanternsecurity.com/bbot

[DBUG] Preset bbot_cli_main: Merging preset "args_preset" into "bbot_cli_main"
[DBUG] Preset bbot_cli_main: Getting baked
[DBUG] Preset bbot_cli_main: Adding module "httpx" of type "scan"
[DBUG] Preset bbot_cli_main: Adding module "csv" of type "output"
[DBUG] Preset bbot_cli_main: Adding module "python" of type "output"
[DBUG] Preset bbot_cli_main: Adding module "stdout" of type "output"
[DBUG] Preset bbot_cli_main: Adding module "json" of type "output"
[DBUG] Preset bbot_cli_main: Adding module "txt" of type "output"
[DBUG] Preset bbot_cli_main: Adding module "speculate" of type "internal"
[DBUG] Preset bbot_cli_main: Adding module "dns" of type "internal"
[DBUG] Preset bbot_cli_main: Adding module "cloud" of type "internal"
[DBUG] Preset bbot_cli_main: Adding module "excavate" of type "internal"
[DBUG] Preset bbot_cli_main: Adding module "aggregate" of type "internal"
[VERB] Creating events from 1 targets
[VERB] 
[VERB] ### MODULES ENABLED ###
[VERB] 
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | Module    | Type     | Needs API Key   | Description                   | Flags                        | Consumed Events      | Produced Events    |
[VERB] +===========+==========+=================+===============================+==============================+======================+====================+
[VERB] | httpx     | scan     | No              | Visit webpages. Many other    | active, cloud-enum, safe,    | OPEN_TCP_PORT, URL,  | HTTP_RESPONSE, URL |
[VERB] |           |          |                 | modules rely on httpx         | social-enum, subdomain-enum, | URL_UNVERIFIED       |                    |
[VERB] |           |          |                 |                               | web-basic                    |                      |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | csv       | output   | No              | Output to CSV                 |                              | *                    |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | json      | output   | No              | Output to Newline-Delimited   |                              | *                    |                    |
[VERB] |           |          |                 | JSON (NDJSON)                 |                              |                      |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | python    | output   | No              | Output via Python API         |                              | *                    |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | stdout    | output   | No              | Output to text                |                              | *                    |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | txt       | output   | No              | Output to text                |                              | *                    |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | cloud     | internal | No              | Tag events by cloud provider, |                              | *                    |                    |
[VERB] |           |          |                 | identify cloud resources like |                              |                      |                    |
[VERB] |           |          |                 | storage buckets               |                              |                      |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | dns       | internal | No              |                               |                              | *                    |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | aggregate | internal | No              | Summarize statistics at the   | passive, safe                |                      |                    |
[VERB] |           |          |                 | end of a scan                 |                              |                      |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | excavate  | internal | No              | Passively extract juicy       | passive                      | HTTP_RESPONSE,       | URL_UNVERIFIED     |
[VERB] |           |          |                 | tidbits from scan data        |                              | RAW_TEXT             |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] | speculate | internal | No              | Derive certain event types    | passive                      | AZURE_TENANT,        | DNS_NAME, FINDING, |
[VERB] |           |          |                 | from others by common sense   |                              | DNS_NAME,            | IP_ADDRESS,        |
[VERB] |           |          |                 |                               |                              | DNS_NAME_UNRESOLVED, | OPEN_TCP_PORT,     |
[VERB] |           |          |                 |                               |                              | HTTP_RESPONSE,       | ORG_STUB           |
[VERB] |           |          |                 |                               |                              | IP_ADDRESS,          |                    |
[VERB] |           |          |                 |                               |                              | IP_RANGE, SOCIAL,    |                    |
[VERB] |           |          |                 |                               |                              | STORAGE_BUCKET, URL, |                    |
[VERB] |           |          |                 |                               |                              | URL_UNVERIFIED,      |                    |
[VERB] |           |          |                 |                               |                              | USERNAME             |                    |
[VERB] +-----------+----------+-----------------+-------------------------------+------------------------------+----------------------+--------------------+
[VERB] Creating events from 1 targets
[VERB] Creating events from 1 whitelist entries
[VERB] Loading word cloud from /root/.bbot/scans/pure_wayne/wordcloud.tsv
[DBUG] Failed to load word cloud from /root/.bbot/scans/pure_wayne/wordcloud.tsv: [Errno 2] No such file or directory: '/root/.bbot/scans/pure_wayne/wordcloud.tsv'
[INFO] Scan with 1 modules seeded with 1 targets (1 in whitelist)
[INFO] Installing the following OS packages: unzip,curl
[DBUG] ansible_run(module=package, args={'name': 'unzip,curl', 'state': 'present'}, ansible_args={'ansible_become': True, 'ansible_become_method': 'sudo'})
The command was not found or was not executable: ansible.
[DBUG] Ansible status: failed
[DBUG] Ansible return code: 127
[DBUG] {
    "event": "verbose",
    "uuid": "b1ce92bc-640d-4573-84cd-ab96e8c5ea18",
    "counter": 1,
    "stdout": "The command was not found or was not executable: ansible.",
    "start_line": 0,
    "end_line": 1,
    "runner_ident": "162057ab-f42a-4deb-84fa-b66b0a5ca755",
    "created": "2024-06-27T15:01:23.863958+00:00"
}
[WARN] Failed to install OS packages (). Recommend installing the following packages manually:
[WARN]  - unzip
[WARN]  - curl
[DBUG] Installing csv - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "csv"
[DBUG] Installing aggregate - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "aggregate"
[DBUG] Installing speculate - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "speculate"
[DBUG] Installing python - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "python"
[DBUG] Installing stdout - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "stdout"
[DBUG] Installing json - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "json"
[DBUG] Installing cloud - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "cloud"
[DBUG] Installing txt - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "txt"
[DBUG] Installing dns - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "dns"
[DBUG] Installing excavate - Preloaded Deps {'modules': [], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [], 'common': []}
[DBUG] No dependency work to do for module "excavate"
[DBUG] Installing httpx - Preloaded Deps {'modules': ['httpx'], 'pip': [], 'pip_constraints': [], 'shell': [], 'apt': [], 'ansible': [{'name': 'Download httpx', 'unarchive': {'src': 'https://github.com/projectdiscovery/httpx/releases/download/v1.2.5/httpx_1.2.5_linux_amd64.zip', 'include': 'httpx', 'dest': '/root/.bbot/tools', 'remote_src': True}, 'become': False}], 'common': []}
[WARN] Skipping dependency install for module "httpx" because it failed previously (--retry-deps to retry or --ignore-failed-deps to ignore)
[ERRR] Failed to install dependencies for 1 modules: httpx (--force to run module anyway)
[TRCE] Traceback (most recent call last):
  File "/root/.local/pipx/venvs/bbot/lib/python3.10/site-packages/bbot/cli.py", line 166, in _main
    await scan._prep()
  File "/root/.local/pipx/venvs/bbot/lib/python3.10/site-packages/bbot/scanner/scanner.py", line 248, in _prep
    await self.load_modules()
  File "/root/.local/pipx/venvs/bbot/lib/python3.10/site-packages/bbot/scanner/scanner.py", line 479, in load_modules
    self._fail_setup(msg)
  File "/root/.local/pipx/venvs/bbot/lib/python3.10/site-packages/bbot/scanner/scanner.py", line 1083, in _fail_setup
    raise ScanError(msg)
bbot.errors.ScanError: Failed to install dependencies for 1 modules: httpx (--force to run module anyway)
TheTechromancer commented 4 days ago

This happens on a fresh ubuntu 22.04 system. The ansible command doesn't appear to be available inside the venv, which is very strange, since it's definitely installed. Activating the venv I was able to confirm that ansible is installed and the command works just fine. But for some reason, BBOT isn't able to see it.

TheTechromancer commented 4 days ago

Fixed in https://github.com/blacklanternsecurity/bbot/pull/1509.