[BUG] TKOSubsScan Error providers-data.csv missing

[Bad3r]

Describe the bug The TKOSubsScan scan fails due to the file providers-data.csv being missing

INFO: [pid 114307] Worker Worker(salt=334832822, workers=1, host=GLaD0S, username=root, pid=114307, sudo_user=bad3r) running   TKOSubsScan(target_file=/tmp/tmpj58rd_um, db_location=/root/.local/recon-pipeline/databases/testing, results_dir=/home/bad3r/tmp/scan-results, exempt_list=, rate=500, interface=enp0s31f6, top_ports=100, ports=)
Error: open None/providers-data.csv: no such file or directory
Usage of /root/.local/recon-pipeline/tools/pipeline-go-workspace/bin/tko-subs:
  -data string
        CSV file containing CMS providers' string for identification (default "providers-data.csv")
  -domain string
        Domains separated by ,
  -domains string
        List of domains to check (default "domains.txt")
  -githubtoken string
        Github personal access token
  -herokuapikey string
        Heroku API key
  -herokuappname string
        Heroku app name
  -herokuusername string
        Heroku username
  -output string
        Output file to save the results (default "output.csv")
  -takeover
        Flag to denote if a vulnerable domain needs to be taken over or not
  -threads int
        Number of threads to run parallel (default 5)
ERROR: [pid 114307] Worker Worker(salt=334832822, workers=1, host=GLaD0S, username=root, pid=114307, sudo_user=bad3r) failed    TKOSubsScan(target_file=/tmp/tmpj58rd_um, db_location=/root/.local/recon-pipeline/databases/testing, results_dir=/home/bad3r/tmp/scan-results, exempt_list=, rate=500, interface=enp0s31f6, top_ports=100, ports=)
Traceback (most recent call last):
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/bad3r/.local/share/virtualenvs/recon-pipeline-B_t2qNnz/lib/python3.8/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/bad3r/git/recon-pipeline/pipeline/recon/web/subdomain_takeover.py", line 132, in run
    self.parse_results()
  File "/home/bad3r/git/recon-pipeline/pipeline/recon/web/subdomain_takeover.py", line 88, in parse_results
    with open(self.output_file, newline="") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/bad3r/tmp/scan-results/tkosubs-results/tkosubs.csv'

To Reproduce Steps to reproduce the behavior:

1. follow installation instructions
2. create a database and attach it
3. run a TKOSubsScan

   scan TKOSubsScan --results-dir /home/bad3r/tmp/scan-results --target secbytes.net --top-ports 100 --rate 500 --interface enp0s31f6 --verbose

Expected behavior the file providers-data.csv should exist before the TKOSubsScan starts

Environment (please complete the following information):

• recon-pipeline version: 0.10.3
• python version: 3.8.3
• OS: Arch Linux

Additional context I am not sure why the parent directory shows as None

open None/providers-data.csv: no such file or directory

initially i noticed the error while running a FullScan

[epi052]

Thanks for the report!

Looking at your traceback, the following line jumps out at me.

...
Error: open None/providers-data.csv: no such file or directory
...

The None preceeding your csv path leads me to believe there's something hinky with the tool definition.

The line where we lookup the csv location is here.

In a good install, we should see the csv here .local/recon-pipeline/tools/pipeline-go-workspace/src/github.com/anshumanbh/tko-subs/providers-data.csv

You found a snag introduced a few commits ago. git_dir was removed from tko-subs.yaml, but we didn't update subdomain_takeover.py to reflect that change.

can you make the following two changes and let me know how it works out?

• pipeline/tools/tko-subs.yaml (add providers line)

  
  installed: false
  dependencies: [go]
  go: &gotool !get_tool_path "{go[path]}"
  path: !join_path [!get_default "{gopath}", bin/tko-subs]
  environ: {"GOPATH": !get_default "{gopath}"}
  providers: !join_path [!get_default "{gopath}", src/github.com/anshumanbh/tko-subs/providers-data.csv]

commands:

• !join [*gotool, get, github.com/anshumanbh/tko-subs]

• pipeline/recon/web/subdomain_takeover.py - modify -data line (line 126)

      command = [
          tools.get("tko-subs").get("path"),
          f"-domain={','.join(domains)}",
          f"-data={tools.get('tko-subs').get('providers')}",
          f"-output={self.output_file}",
      ]

[Bad3r]

that fixed it thanks!

[epi052]

Thanks for checking the fix! I'm going to reopen this in order to track the problem until i get a PR in to fix it.