Errors during nmap XML parsing

[sebrink]

I was attempting to run WitnessMe with a large nmap XML file for the top 1000 ports in a certain range. When passing this to WitnessMe (using both pipx and docker) I received the following error:

# docker run -it --entrypoint=/bin/sh -v $(pwd):/transfer a45ee02a44c1
$ witnessme screenshot /transfer/top1000.xml
[witnessme.screenshot] INFO - Starting scan 593de326-9a88-4ee1-9701-305af0bfb66b
Task exception was never retrieved
future: <Task finished name='Task-3' coro=<HeadlessChromium.target_producer() done, defined at /home/appuser/.local/lib/python3.8/site-packages/witnessme/headlessbrowser.py:64> exception=KeyError('port')>
Traceback (most recent call last):
  File "/home/appuser/.local/lib/python3.8/site-packages/witnessme/headlessbrowser.py", line 66, in target_producer
    for url in generated_targets:
  File "/home/appuser/.local/lib/python3.8/site-packages/witnessme/parsers.py", line 198, in generate
    for url in generated_urls:
  File "/home/appuser/.local/lib/python3.8/site-packages/witnessme/parsers.py", line 85, in __enter__
    xmltodict.parse(
  File "/home/appuser/.local/lib/python3.8/site-packages/xmltodict.py", line 325, in parse
    parser.ParseFile(xml_input)
  File "/usr/src/python/Modules/pyexpat.c", line 461, in EndElement
  File "/home/appuser/.local/lib/python3.8/site-packages/xmltodict.py", line 126, in endElement
    should_continue = self.item_callback(self.path, item)
  File "/home/appuser/.local/lib/python3.8/site-packages/witnessme/parsers.py", line 108, in parser_callback
    ports = item["ports"]["port"]
KeyError: 'port'

I was able to use gowitness and it parsed the file normally, so I believe this has something to do with how WitnessMe handles nmap XML parsing.

[0xArt3mis]

The problem is that it raises an error when parsing the xml if no port is open but e.g. a filtered port is shown in in the xml of the host. The parsers.py does not handle this correctly, therefore an error is raised.

For me the issue was fixed when changing the following code of parsers.py

class NmapParser(XmlParser):
    def __init__(self, file_path):
        super().__init__(file_path)
        self.item_depth = 2

    def parser_callback(self, path, item):
        if isinstance(item, OrderedDict):
            if "address" in item.keys() and "ports" in item.keys():
                address = item["address"]["@addr"]
                ports = item["ports"]["port"]

                # If there's only a single port discovered, ports will be an OrderedDict
                if isinstance(ports, OrderedDict):
                    ports = [ports]

                for port in ports:
                    if port["@protocol"] == "tcp" and port["state"]["@state"] == "open":
                        service = port["service"].get("@name")
                        port_number = port["@portid"]
                        if "ssl" in service or service == "https":
                            self.urls.add(f"https://{address}:{port_number}")
                        elif service == "http-alt" or service == "http":
                            self.urls.add(f"http://{address}:{port_number}")
        return True

to

class NmapParser(XmlParser):
    def __init__(self, file_path):
        super().__init__(file_path)
        self.item_depth = 2

    def parser_callback(self, path, item):
        if isinstance(item, OrderedDict):
            if "address" in item.keys() and "ports" in item.keys():
                if "port" in item["ports"].keys() and "@addr" in item["address"].keys():
                    address = item["address"]["@addr"]
                    ports = item["ports"]["port"]

                # If there's only a single port discovered, ports will be an OrderedDict
                    if isinstance(ports, OrderedDict):
                        ports = [ports]

                    for port in ports:
                        if port["@protocol"] == "tcp" and port["state"]["@state"] == "open":
                            service = port["service"].get("@name")
                            tunnel = port["service"].get("@tunnel")
                            port_number = port["@portid"]
                            if "ssl" in service or service == "https" or (service == "http" and tunnel == "ssl") or \
                                    (service == "http-proxy" and tunnel == "ssl"):
                                self.urls.add(f"https://{address}:{port_number}")
                            elif service == "http-alt" or service == "http" or service == "http-proxy":
                                self.urls.add(f"http://{address}:{port_number}")
        return True

The following check on line number 107 was added to the previous code which fixed the issue for me:

if "port" in item["ports"].keys() and "@addr" in item["address"].keys():

Does this fix the issue for you as well?

Note that in the port section I've additionally added triggers for ports which were not detected correctly in my nmap file. Since a "http" service which is using ssl on port 443 was not detected with the preconfigured triggers. Some ports were detected as "http-proxy" service, so I've added that as well.

[7MinSec]

Hi,

Just wanted to subscribe to this thread as I'm having very similar output issues to @sebrink's scan. @CT-H00K I'd be interested in trying your fix. I've got Witnessme running in Docker and I'm a Docker novice so if you wouldn't mind spelling out the instructions a bit for tweaking this .py file I can give it a swing.

Thanks, Brian

[0xArt3mis]

Hi,

if you have pulled the image with:

docker pull byt3bl33d3r/witnessme

You can check the $IMAGEID with the following command:

docker image ls

Use the IMAGE ID to drop into a root shell with the following command:

docker run --user=root -it --entrypoint=/bin/bash $IMAGEID

To edit files you would need an editor within the docker, which might not be installed. Therefore just install your editor within the docker using apt install vim

Within the container you need to edit the parsers.py file as mentioned above, might be somewhere at:

/home/appuser/src/witnessme/witnessme/parsers.py

Note that you need to move all lines below the change with 4 spaces except the return True in the last line.

After you made the changes to the file, drop into an appuser shell with su appuser and run witnessme. I've not tested it with docker, so could be that I missed something here.

The change will not be persistent but will be enough for testing the fix. For persistence you would need to e.g. map the file into the container from your host.

[byt3bl33d3r]

@CT-H00K @braimee this was fixed in https://github.com/byt3bl33d3r/WitnessMe/commit/2fbd06c8c39e10d09322597eb8a5e89fb9ac3cbd

Thanks

[7MinSec]

Thanks @byt3bl33d3r and @CT-H00K, got my first scan done today and she's a beaut!