search

jhaddix / domain

Setup script for Regon-ng
913 stars 242 forks source link

Update to support python3 and recon-ng v5 #34

Open neubauek opened 4 years ago

neubauek commented 4 years ago

No longer compatible with main branch of recon-ng as recon-ng has moved to python3 and has updated their code, which breaks some references in enumall.py.

Oneiroi commented 4 years ago

Also facing this issue, Having corrected the Python3 incompatibilities of enumall.py I notice that recon-ng v5 has also changes some of their python API:

Traceback (most recent call last):
  File "./enumall.py", line 98, in <module>
    run_recon(domainList, bruteforceList)
  File "./enumall.py", line 46, in run_recon
    reconb.init_workspace(wspace)
AttributeError: 'Recon' object has no attribute 'init_workspace'

This too will need to be considered in addition to the python3 syntax fix (which is only a few parenthesis additions to enumall.py)

Oneiroi commented 4 years ago

Diff for enumwall.py to make python3 compatible, call the relocated _init_workspaces

diff --git a/enumall.py b/enumall.py
index 15d0e55..98cb2ef 100755
--- a/enumall.py
+++ b/enumall.py
@@ -43,7 +43,7 @@ def run_recon(domains, bruteforce):
        wspace = domains[0]+stamp

        reconb = base.Recon(base.Mode.CLI)
-       reconb.init_workspace(wspace)
+       reconb._init_workspace(wspace)
        reconb.onecmd("TIMEOUT=100")
        module_list = ["recon/domains-hosts/bing_domain_web", "recon/domains-hosts/google_site_web", "recon/domains-hosts/netcraft", "recon/domains-hosts/shodan_hostname", "recon/netblocks-companies/whois_orgs", "recon/hosts-hosts/resolve"]

@@ -80,7 +80,7 @@ parser.add_argument("-p", dest="permlist", type=argparse.FileType('r'), help="in
 args = parser.parse_args()

 if args.runAltDns and not altDnsPath:
-       print "Error: no altDns path specified, please download from: https://github.com/infosec-au/altdns"
+       print ("Error: no altDns path specified, please download from: https://github.com/infosec-au/altdns")
        exit(0)

 domainList = []
@@ -103,6 +103,6 @@ if args.runAltDns:
        subdomains = os.path.join(os.getcwd(), workspace+".lst")
        permList = args.permlist.name if args.permlist else os.path.join(altDnsPath,"words.txt")
        output = os.path.join(os.getcwd(),workspace+"_output.txt")
-       print "running alt-dns... please be patient :) results will be displayed in "+output
+       print("running alt-dns... please be patient :) results will be displayed in "+output)

This is not (yet) working for me however, getting:

Traceback (most recent call last):
  File "./enumall.py", line 98, in <module>
    run_recon(domainList, bruteforceList)
  File "./enumall.py", line 46, in run_recon
    reconb._init_workspace(wspace)
  File "/usr/share/recon-ng/recon/core/base.py", line 192, in _init_workspace
    self._create_db()
  File "/usr/share/recon-ng/recon/core/base.py", line 229, in _create_db
    self.query('CREATE TABLE IF NOT EXISTS domains (domain TEXT, module TEXT)')
  File "/usr/share/recon-ng/recon/core/framework.py", line 356, in query
    return self._query(path, *args, **kwargs)
  File "/usr/share/recon-ng/recon/core/framework.py", line 360, in _query
    self.debug(f"DATABASE => {path}")
  File "/usr/share/recon-ng/recon/core/framework.py", line 288, in debug
    if self._global_options['verbosity'] >= 2:
  File "/usr/share/recon-ng/recon/core/framework.py", line 40, in __getitem__
    return super(Options, self).__getitem__(name)
KeyError: 'VERBOSITY'

(iterative break/fix is in progress)

Oneiroi commented 4 years ago

UPdated diff

diff --git a/enumall.py b/enumall.py
index 15d0e55..4ed828f 100755
--- a/enumall.py
+++ b/enumall.py
@@ -43,7 +43,8 @@ def run_recon(domains, bruteforce):
        wspace = domains[0]+stamp

        reconb = base.Recon(base.Mode.CLI)
-       reconb.init_workspace(wspace)
+       reconb._global_options['verbosity'] = 1
+       reconb._init_workspace(wspace)
        reconb.onecmd("TIMEOUT=100")
        module_list = ["recon/domains-hosts/bing_domain_web", "recon/domains-hosts/google_site_web", "recon/domains-hosts/netcraft", "recon/domains-hosts/shodan_hostname", "recon/netblocks-companies/whois_orgs", "recon/hosts-hosts/resolve"]

@@ -80,7 +81,7 @@ parser.add_argument("-p", dest="permlist", type=argparse.FileType('r'), help="in
 args = parser.parse_args()

 if args.runAltDns and not altDnsPath:
-       print "Error: no altDns path specified, please download from: https://github.com/infosec-au/altdns"
+       print ("Error: no altDns path specified, please download from: https://github.com/infosec-au/altdns")
        exit(0)

 domainList = []
@@ -103,6 +104,6 @@ if args.runAltDns:
        subdomains = os.path.join(os.getcwd(), workspace+".lst")
        permList = args.permlist.name if args.permlist else os.path.join(altDnsPath,"words.txt")
        output = os.path.join(os.getcwd(),workspace+"_output.txt")
-       print "running alt-dns... please be patient :) results will be displayed in "+output

current issue is that Recon.do_load() method no longer exists;

Traceback (most recent call last):
  File "./enumall.py", line 99, in <module>
    run_recon(domainList, bruteforceList)
  File "./enumall.py", line 53, in run_recon
    run_module(reconb, module, domain)
  File "./enumall.py", line 36, in run_module
    x = reconBase.do_load(module)
AttributeError: 'Recon' object has no attribute 'do_load'

Seeing if I can find the method equivalent with the Recon object, however searches for the keyword load do not immediatly make it obvious as to where this may be.

Oneiroi commented 4 years ago

(if anyone is versed with recon-ng v5 python framework feel free to point me in the correct direction, currently I'm itterative break & fixing in a limited amount of time (and may have to abandon the effort)

update: tweeted to the recon-ng author: https://twitter.com/icleus/status/1220038591285485577 with luck I can find or be pointed to the correct method to use.

Oneiroi commented 4 years ago

The story so far ...

python3 ./enumall.py domain.com
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS domains (domain TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS companies (company TEXT, description TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS netblocks (netblock TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS locations (latitude TEXT, longitude TEXT, street_address TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS vulnerabilities (host TEXT, reference TEXT, example TEXT, publish_date TEXT, category TEXT, status TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS ports (ip_address TEXT, host TEXT, port TEXT, protocol TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS hosts (host TEXT, ip_address TEXT, region TEXT, country TEXT, latitude TEXT, longitude TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS contacts (first_name TEXT, middle_name TEXT, last_name TEXT, email TEXT, title TEXT, region TEXT, country TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS credentials (username TEXT, password TEXT, hash TEXT, type TEXT, leak TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS leaks (leak_id TEXT, description TEXT, source_refs TEXT, leak_type TEXT, title TEXT, import_date TEXT, leak_date TEXT, attackers TEXT, num_entries TEXT, score TEXT, num_domains_affected TEXT, attack_method TEXT, target_industries TEXT, password_hash TEXT, password_type TEXT, targets TEXT, media_refs TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS pushpins (source TEXT, screen_name TEXT, profile_name TEXT, profile_url TEXT, media_url TEXT, thumb_url TEXT, message TEXT, latitude TEXT, longitude TEXT, time TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS profiles (username TEXT, resource TEXT, url TEXT, category TEXT, notes TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS repositories (name TEXT, owner TEXT, description TEXT, resource TEXT, category TEXT, url TEXT, module TEXT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => CREATE TABLE IF NOT EXISTS dashboard (module TEXT PRIMARY KEY, runs INT)
[*] DATABASE => /home/dbusby/.recon-ng/workspaces/domain.com35:10-01_23_2020/data.db
[*] QUERY => PRAGMA user_version = 8
[*] Updating index file...
[!] Invalid command: TIMEOUT=100.
/home/dbusby/.recon-ng/modules
/home/dbusby/.recon-ng/modules/
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/recon-ng/recon/core/base.py", line 452, in _load_module
    self._loaded_modules[mod_dispname] = sys.modules[mod_loadname].Module(mod_dispname)
  File "/usr/share/recon-ng/recon/core/module.py", line 25, in __init__
    self.meta = self._merge_dicts(self.meta, self._parse_frontmatter())
  File "/usr/share/recon-ng/recon/core/module.py", line 67, in _parse_frontmatter
    with open(abs_path) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/recon/domains-hosts/bing_domain_web.py'
------------------------------------------------------------
[!] Module '/recon/domains-hosts/bing_domain_web' disabled.

Scratching my head now trying to figure out why it's not using the dirpath passed to load the module file...

Current DIFF (I plan to clean some of this up):

diff --git a/enumall.py b/enumall.py
index 15d0e55..a605ddc 100755
--- a/enumall.py
+++ b/enumall.py
@@ -33,17 +33,26 @@ if altDnsPath:

 def run_module(reconBase, module, domain):
-       x = reconBase.do_load(module)
-       x.do_set("SOURCE " + domain)
-       x.do_run(None)
-
+    #recon-ng v5 stores in a user home hidden directory
+       print(reconBase.mod_path)
+       modulePrefix = os.path.expanduser("~/.recon-ng/modules/")
+       print(modulePrefix)
+       if ".py" not in module:
+               module += ".py"
+       x = reconBase._load_module(filename=module, dirpath=modulePrefix)
+       if x:
+               x.do_set("SOURCE " + domain)
+               x.do_run(None)
+       else:
+               sys.exit()

 def run_recon(domains, bruteforce):
        stamp = datetime.datetime.now().strftime('%M:%H-%m_%d_%Y')
        wspace = domains[0]+stamp

        reconb = base.Recon(base.Mode.CLI)
-       reconb.init_workspace(wspace)
+       reconb._global_options['verbosity'] = 2
+       reconb._init_workspace(wspace)
        reconb.onecmd("TIMEOUT=100")
        module_list = ["recon/domains-hosts/bing_domain_web", "recon/domains-hosts/google_site_web", "recon/domains-hosts/netcraft", "recon/domains-hosts/shodan_hostname", "recon/netblocks-companies/whois_orgs", "recon/hosts-hosts/resolve"]

@@ -80,7 +89,7 @@ parser.add_argument("-p", dest="permlist", type=argparse.FileType('r'), help="in
 args = parser.parse_args()

 if args.runAltDns and not altDnsPath:
-       print "Error: no altDns path specified, please download from: https://github.com/infosec-au/altdns"
+       print ("Error: no altDns path specified, please download from: https://github.com/infosec-au/altdns")
        exit(0)

 domainList = []
@@ -103,6 +112,6 @@ if args.runAltDns:
        subdomains = os.path.join(os.getcwd(), workspace+".lst")
        permList = args.permlist.name if args.permlist else os.path.join(altDnsPath,"words.txt")
        output = os.path.join(os.getcwd(),workspace+"_output.txt")
-       print "running alt-dns... please be patient :) results will be displayed in "+output
+       print("running alt-dns... please be patient :) results will be displayed in "+output)
        # python altdns.py -i subdomainsList -o data_output -w permutationsList -r -s results_output.txt
        os.system('%s -i %s -o data_output -w %s -r -s %s' % (altCmd, subdomains, permList,output))

Hopefully this may spur $someone into thinking, "I know how to fix that" whom is more familiar with the code bases, until that time I'll try to work on this when I have a moment to do so.

richard1230 commented 4 years ago

I have the same problem with you，do you resolve this?

drtychai commented 3 years ago

Resolved in #38