Hydra Output to JSOn file incomplete

Hydra version: Hydra v9.5 (c) 2023 by van Hauser/THC & David Maciejak Os version: Linux version 5.15.0-107-generic (buildd@lcy02-amd64-012) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 Python version: Python 3.10.12

Code:

def checkfunc():
    tmppath_o = f"/tmp/{uuid.uuid4()}"
    cmd_ = f"""hydra -I -u -q -b json -4 {part_user} -P {self.passwordspath} -s {port} {ip} {serivcename} -t 64 -o {tmppath_o}"""
    self._thislogclass.print_and_log("hydra cmd: "+ cmd_)
    command_list = shlex.split(cmd_)
    try:
        result = subprocess.run(command_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=600)
    except Exception as e:
        error_str = str(traceback.format_exc())
        print(error_str)
    if os.path.exists(tmppath_o):
        with open(tmppath_o, "r") as f:
            z = f.read()
        try:
            z = json.loads(z)
        except Exception as e:
           error_str = str(traceback.format_exc())
           print("data format error:", z)

with concurrent.futures.ThreadPoolExecutor(max_workers=(100)) as executor:
      futures = []
      for onetask in all_tasks:
          futures.append(executor.submit(self.checkfunc, onetask ))
      alres = [future.result() for future in futures]

And you will get this except:

{ "generator": {
        "software": "Hydra", "version": "v9.5", "built": "2024-08-04 12:32:02",
        "server": "192.168.2.17", "service": "ssh", "jsonoutputversion": "1.00",
        "commandline": "hydra -I -u -q -b json -4 -L /opt/seccmdb/scanner/users.txt -P /opt/seccmdb/scanner/passwords.txt -s 22 -t 64 -o /tmp/14f2511a-eb1b-4856-b2b0-562106da7d34 192.168.2.17 ssh"
        },
"results": [

As you see ,the file is a invalid JSON file , so python3 load it failed!
But excecute Command: hydra -I -u -q -b json -4 -L /opt/seccmdb/scanner/users.txt -P /opt/seccmdb/scanner/passwords.txt -s 22 -t 64 -o /tmp/14f2511a-eb1b-4856-b2b0-562106da7d34 192.168.2.17 ssh alone , the output is fine, so i need your help !

The issue you're encountering happens because the JSON output generated by hydra is not correctly formatted when run through subprocess.run(). However, when you execute the same command directly in the terminal, the JSON output is valid.

This discrepancy could be due to several reasons:

Interleaved Output: Even with the -q (quiet) option, hydra may still print status messages, debug logs, or progress information that could interfere with the JSON output when run via subprocess.run().
Race Conditions: When using concurrent.futures.ThreadPoolExecutor, multiple hydra instances running concurrently might affect the output, especially if writing to the same file or if there's shared stdout/stderr.

Potential Solutions

Solution 1: Separate `stdout` and `stderr` Processing

Ensure that stdout and stderr are processed separately to avoid mixing JSON output with error messages or other logs:

import subprocess
import json
import shlex
import uuid
import os
import traceback

def checkfunc(part_user, self, port, ip, servicename):
    tmppath_o = f"/tmp/{uuid.uuid4()}"
    cmd_ = f"""hydra -I -u -q -b json -4 {part_user} -P {self.passwordspath} -s {port} {ip} {servicename} -t 64 -o {tmppath_o}"""
    self._thislogclass.print_and_log("Hydra command: " + cmd_)
    command_list = shlex.split(cmd_)
    try:
        result = subprocess.run(command_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=600)
        if result.returncode != 0:
            print(f"Hydra error: {result.stderr.decode()}")
    except Exception:
        error_str = traceback.format_exc()
        print(error_str)

    # Read JSON output from the temporary file
    if os.path.exists(tmppath_o):
        with open(tmppath_o, "r") as f:
            output = f.read()
        try:
            # Remove non-JSON content if present
            json_start = output.find('{')
            if json_start != -1:
                output = output[json_start:]
            data = json.loads(output)
            return data
        except Exception:
            error_str = traceback.format_exc()
            print("JSON parsing error:", output)

Solution 2: Use a Dedicated Temporary File for Output

Instead of relying on stdout, let hydra handle output directly into a temporary file and read from it afterward. Ensure you parse only the valid JSON content.

Solution 3: Validate and Clean Up Output

If hydra includes unexpected logs or status updates before the JSON output, you can strip these out before parsing:

if os.path.exists(tmppath_o):
    with open(tmppath_o, "r") as f:
        output = f.read()
    # Find the first occurrence of '{' to detect the start of the JSON
    json_start = output.find('{')
    if json_start != -1:
        try:
            # Extract only the JSON part
            output = output[json_start:]
            data = json.loads(output)
            return data
        except Exception:
            error_str = traceback.format_exc()
            print("Data format error:", output)
    else:
        print("No valid JSON structure found:", output)

Explanation

Filtering the Output: The code searches for the first occurrence of {, as JSON objects always start with this character. If hydra outputs additional logs before the actual JSON, they will be ignored.
Separate stdout/stderr Handling: By handling stdout and stderr separately, you prevent any error messages from corrupting the JSON output.

These solutions should help address the issue by ensuring that only valid JSON content is processed, even when hydra outputs extra data. Give these adjustments a try and see if they resolve your problem.

vanhauser-thc / thc-hydra