search

lanmaster53 / recon-ng

Open Source Intelligence gathering tool aimed at reducing the time spent harvesting information from open sources.
GNU General Public License v3.0
3.87k stars 623 forks source link

REQUIREMENTS file missing two modules #46

Closed BBhacKing closed 4 years ago

BBhacKing commented 4 years ago

After a fresh install of 5.0.1 on Ubuntu 18.04, including running pip3 install -r REQUIREMENTS, the framework complains at startup about two missing modules. The missing modules are not listed in the REQUIREMENTS file.

[!] Module 'recon/domains-credentials/pwnedlist/domain_creds' disabled. Dependency required: ''pyaes''.
...
[!] Module 'recon/domains-contacts/metacrawler' disabled. Dependency required: ''PyPDF3''.

Adding 'pyPDF3' and 'pyaes' to the REQUIREMENTS file and re-running pip3 install -r REQUIREMENTS resolved the issue.

lanmaster53 commented 4 years ago

REQUIREMENTS isn't supposed to include these dependencies. The REQUIREMENTS file is only for framework dependencies. The dependencies you mention above are for modules that you had preinstalled (which shouldn't be happening if you are installing from the marketplace). When installing a module from the marketplace, the user is responsible for installing its dependencies, which are listed in the marketplace info command and annotated in the marketplace search results by an "*" under the "D" column. This reduces the number of dependencies for framework and for those that don't use the modules that require them. This is addressed in the wiki at https://github.com/lanmaster53/recon-ng/wiki/Features#module-marketplace.

BBhacKing commented 4 years ago

I don't have any modules preinstalled.

To double-check, I started up a VM that's never had recon-ng on it, cloned 5.1.0 from the repo, installed the dependencies, then installed the marketplace modules. I see the same errors. Details below.

marketplace install all included four modules that it immediately complained about missing dependencies. Summary below followed by full log of installing from REQUIREMENTS, purging ~/.recon-ng directory, starting recon-ng, seeing no modules installed, running marketplace install all, seeing the messages about missing dependencies. The list of modules installed by the marketplace command includes the ones that the framework immediately complains about.

Missing Dependencies Extracted from recon-ng Message:

[!] Module 'recon/domains-contacts/metacrawler' disabled. Dependency required: ''PyPDF3''.
[!] Module 'recon/domains-credentials/pwnedlist/account_creds' disabled. Dependency required: ''pyaes''.
[!] Module 'recon/domains-credentials/pwnedlist/domain_creds' disabled. Dependency required: ''pyaes''.
[!] Module 'recon/contacts-contacts/abc' disabled. Dependency required: ''bs4''.

Full log from my terminal

user@ubuntu ~/recon-ng % pip3 install -r REQUIREMENTS 
Collecting pyyaml (from -r REQUIREMENTS (line 2))
Collecting dnspython (from -r REQUIREMENTS (line 3))
  Using cached https://files.pythonhosted.org/packages/ec/d3/3aa0e7213ef72b8585747aa0e271a9523e713813b9a20177ebe1e939deb0/dnspython-1.16.0-py2.py3-none-any.whl
Collecting lxml (from -r REQUIREMENTS (line 4))
  Using cached https://files.pythonhosted.org/packages/ec/be/5ab8abdd8663c0386ec2dd595a5bc0e23330a0549b8a91e32f38c20845b6/lxml-4.4.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting mechanize (from -r REQUIREMENTS (line 5))
  Using cached https://files.pythonhosted.org/packages/88/04/178c60f59e8e0050c594429b76029cfaa1f0afd4bdbaaad195141fbea9bb/mechanize-0.4.4-py2.py3-none-any.whl
Collecting requests (from -r REQUIREMENTS (line 6))
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting flask (from -r REQUIREMENTS (line 8))
  Using cached https://files.pythonhosted.org/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl
Collecting flask-restful (from -r REQUIREMENTS (line 9))
  Using cached https://files.pythonhosted.org/packages/17/44/6e490150ee443ca81d5f88b61bb4bbb133d44d75b0b716ebe92489508da4/Flask_RESTful-0.3.7-py2.py3-none-any.whl
Collecting flasgger (from -r REQUIREMENTS (line 10))
  Using cached https://files.pythonhosted.org/packages/23/c0/68b331d0d6ab787b8e7d027f01dc7771040545200ee8e1e88a98b516d29b/flasgger-0.9.3-py2.py3-none-any.whl
Collecting dicttoxml (from -r REQUIREMENTS (line 11))
Collecting XlsxWriter (from -r REQUIREMENTS (line 12))
  Using cached https://files.pythonhosted.org/packages/25/88/a38f35b00ce4dd166a20e1a0a25e438e19e332e680df52af4aeac0df0f03/XlsxWriter-1.2.2-py2.py3-none-any.whl
Collecting unicodecsv (from -r REQUIREMENTS (line 13))
Collecting rq (from -r REQUIREMENTS (line 14))
  Using cached https://files.pythonhosted.org/packages/8a/22/e49d976266012d5e800d5500b3eaabbb40d286d67cea775a4b87d1de52b8/rq-1.1.0-py2.py3-none-any.whl
Collecting html5lib>=0.999999999 (from mechanize->-r REQUIREMENTS (line 5))
  Using cached https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->-r REQUIREMENTS (line 6))
  Using cached https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests->-r REQUIREMENTS (line 6))
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->-r REQUIREMENTS (line 6))
  Using cached https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->-r REQUIREMENTS (line 6))
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting Jinja2>=2.10.1 (from flask->-r REQUIREMENTS (line 8))
  Using cached https://files.pythonhosted.org/packages/65/e0/eb35e762802015cab1ccee04e8a277b03f1d8e53da3ec3106882ec42558b/Jinja2-2.10.3-py2.py3-none-any.whl
Collecting Werkzeug>=0.15 (from flask->-r REQUIREMENTS (line 8))
  Using cached https://files.pythonhosted.org/packages/ce/42/3aeda98f96e85fd26180534d36570e4d18108d62ae36f87694b476b83d6f/Werkzeug-0.16.0-py2.py3-none-any.whl
Collecting itsdangerous>=0.24 (from flask->-r REQUIREMENTS (line 8))
  Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting click>=5.1 (from flask->-r REQUIREMENTS (line 8))
  Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Collecting six>=1.3.0 (from flask-restful->-r REQUIREMENTS (line 9))
  Using cached https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Collecting aniso8601>=0.82 (from flask-restful->-r REQUIREMENTS (line 9))
  Using cached https://files.pythonhosted.org/packages/eb/e4/787e104b58eadc1a710738d4e418d7e599e4e778e52cb8e5d5ef6ddd5833/aniso8601-8.0.0-py2.py3-none-any.whl
Collecting pytz (from flask-restful->-r REQUIREMENTS (line 9))
  Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Collecting mistune (from flasgger->-r REQUIREMENTS (line 10))
  Using cached https://files.pythonhosted.org/packages/09/ec/4b43dae793655b7d8a25f76119624350b4d65eb663459eb9603d7f1f0345/mistune-0.8.4-py2.py3-none-any.whl
Collecting jsonschema<3.0.0 (from flasgger->-r REQUIREMENTS (line 10))
  Using cached https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting redis>=3.0.0 (from rq->-r REQUIREMENTS (line 14))
  Using cached https://files.pythonhosted.org/packages/32/ae/28613a62eea0d53d3db3147f8715f90da07667e99baeedf1010eb400f8c0/redis-3.3.11-py2.py3-none-any.whl
Collecting webencodings (from html5lib>=0.999999999->mechanize->-r REQUIREMENTS (line 5))
  Using cached https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10.1->flask->-r REQUIREMENTS (line 8))
  Using cached https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: pyyaml, dnspython, lxml, webencodings, six, html5lib, mechanize, urllib3, idna, certifi, chardet, requests, MarkupSafe, Jinja2, Werkzeug, itsdangerous, click, flask, aniso8601, pytz, flask-restful, mistune, jsonschema, flasgger, dicttoxml, XlsxWriter, unicodecsv, redis, rq
Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 Werkzeug-0.16.0 XlsxWriter-1.2.2 aniso8601-8.0.0 certifi-2019.9.11 chardet-3.0.4 click-7.0 dicttoxml-1.7.4 dnspython-1.16.0 flasgger-0.9.3 flask-1.1.1 flask-restful-0.3.7 html5lib-1.0.1 idna-2.8 itsdangerous-1.1.0 jsonschema-3.1.1 lxml-4.4.1 mechanize-0.4.4 mistune-0.8.4 pytz-2019.3 pyyaml-5.1.2 redis-3.3.11 requests-2.22.0 rq-1.1.0 six-1.13.0 unicodecsv-0.14.1 urllib3-1.25.6 webencodings-0.5.1
user@ubuntu ~/recon-ng % rm -r ~/.recon-ng
user@ubuntu ~/recon-ng % ./recon-ng

    _/_/_/    _/_/_/_/    _/_/_/    _/_/_/    _/      _/            _/      _/    _/_/_/
   _/    _/  _/        _/        _/      _/  _/_/    _/            _/_/    _/  _/       
  _/_/_/    _/_/_/    _/        _/      _/  _/  _/  _/  _/_/_/_/  _/  _/  _/  _/  _/_/_/
 _/    _/  _/        _/        _/      _/  _/    _/_/            _/    _/_/  _/      _/ 
_/    _/  _/_/_/_/    _/_/_/    _/_/_/    _/      _/            _/      _/    _/_/_/    

                                          /\
                                         / \\ /\
    Sponsored by...               /\  /\/  \\V  \/\
                                 / \\/ // \\\\\ \\ \/\
                                // // BLACK HILLS \/ \\
                               www.blackhillsinfosec.com

                  ____   ____   ____   ____ _____ _  ____   ____  ____
                 |____] | ___/ |____| |       |   | |____  |____ |
                 |      |   \_ |    | |____   |   |  ____| |____ |____
                                   www.practisec.com

                      [recon-ng v5.1.0, Tim Tomes (@lanmaster53)]                       

[*] No modules enabled/installed.

[recon-ng][default] > marketplace install all
[*] Module installed: discovery/info_disclosure/cache_snoop
[*] Module installed: discovery/info_disclosure/interesting_files
[*] Module installed: exploitation/injection/command_injector
[*] Module installed: exploitation/injection/xpath_bruter
[*] Module installed: import/csv_file
[*] Module installed: import/list
[*] Module installed: import/nmap
[*] Module installed: recon/companies-contacts/bing_linkedin_cache
[*] Module installed: recon/companies-contacts/pen
[*] Module installed: recon/companies-domains/pen
[*] Module installed: recon/companies-domains/viewdns_reverse_whois
[*] Module installed: recon/companies-multi/github_miner
[*] Module installed: recon/companies-multi/shodan_org
[*] Module installed: recon/companies-multi/whois_miner
[*] Module installed: recon/contacts-contacts/abc
[*] Module installed: recon/contacts-contacts/mailtester
[*] Module installed: recon/contacts-contacts/mangle
[*] Module installed: recon/contacts-contacts/unmangle
[*] Module installed: recon/contacts-credentials/hibp_breach
[*] Module installed: recon/contacts-credentials/hibp_paste
[*] Module installed: recon/contacts-credentials/scylla
[*] Module installed: recon/contacts-domains/migrate_contacts
[*] Module installed: recon/contacts-profiles/fullcontact
[*] Module installed: recon/credentials-credentials/adobe
[*] Module installed: recon/credentials-credentials/bozocrack
[*] Module installed: recon/credentials-credentials/hashes_org
[*] Module installed: recon/domains-companies/pen
[*] Module installed: recon/domains-contacts/metacrawler
[*] Module installed: recon/domains-contacts/pen
[*] Module installed: recon/domains-contacts/pgp_search
[*] Module installed: recon/domains-contacts/whois_pocs
[*] Module installed: recon/domains-credentials/pwnedlist/account_creds
[*] Module installed: recon/domains-credentials/pwnedlist/api_usage
[*] Module installed: recon/domains-credentials/pwnedlist/domain_creds
[*] Module installed: recon/domains-credentials/pwnedlist/domain_ispwned
[*] Module installed: recon/domains-credentials/pwnedlist/leak_lookup
[*] Module installed: recon/domains-credentials/pwnedlist/leaks_dump
[*] Module installed: recon/domains-credentials/scylla
[*] Module installed: recon/domains-domains/brute_suffix
[*] Module installed: recon/domains-hosts/binaryedge
[*] Module installed: recon/domains-hosts/bing_domain_api
[*] Module installed: recon/domains-hosts/bing_domain_web
[*] Module installed: recon/domains-hosts/brute_hosts
[*] Module installed: recon/domains-hosts/builtwith
[*] Module installed: recon/domains-hosts/certificate_transparency
[*] Module installed: recon/domains-hosts/findsubdomains
[*] Module installed: recon/domains-hosts/google_site_web
[*] Module installed: recon/domains-hosts/hackertarget
[*] Module installed: recon/domains-hosts/mx_spf_ip
[*] Module installed: recon/domains-hosts/netcraft
[*] Module installed: recon/domains-hosts/shodan_hostname
[*] Module installed: recon/domains-hosts/ssl_san
[*] Module installed: recon/domains-hosts/threatcrowd
[*] Module installed: recon/domains-hosts/threatminer
[*] Module installed: recon/domains-vulnerabilities/ghdb
[*] Module installed: recon/domains-vulnerabilities/xssed
[*] Module installed: recon/domains-vulnerabilities/xssposed
[*] Module installed: recon/hosts-domains/migrate_hosts
[*] Module installed: recon/hosts-hosts/bing_ip
[*] Module installed: recon/hosts-hosts/ipinfodb
[*] Module installed: recon/hosts-hosts/ipstack
[*] Module installed: recon/hosts-hosts/resolve
[*] Module installed: recon/hosts-hosts/reverse_resolve
[*] Module installed: recon/hosts-hosts/ssltools
[*] Module installed: recon/hosts-hosts/virustotal
[*] Module installed: recon/hosts-locations/migrate_hosts
[*] Module installed: recon/hosts-ports/binaryedge
[*] Module installed: recon/hosts-ports/shodan_ip
[*] Module installed: recon/locations-locations/geocode
[*] Module installed: recon/locations-locations/reverse_geocode
[*] Module installed: recon/locations-pushpins/flickr
[*] Module installed: recon/locations-pushpins/shodan
[*] Module installed: recon/locations-pushpins/twitter
[*] Module installed: recon/locations-pushpins/youtube
[*] Module installed: recon/netblocks-companies/whois_orgs
[*] Module installed: recon/netblocks-hosts/reverse_resolve
[*] Module installed: recon/netblocks-hosts/shodan_net
[*] Module installed: recon/netblocks-hosts/virustotal
[*] Module installed: recon/netblocks-ports/census_2012
[*] Module installed: recon/netblocks-ports/censysio
[*] Module installed: recon/ports-hosts/migrate_ports
[*] Module installed: recon/profiles-contacts/bing_linkedin_contacts
[*] Module installed: recon/profiles-contacts/dev_diver
[*] Module installed: recon/profiles-contacts/github_users
[*] Module installed: recon/profiles-profiles/namechk
[*] Module installed: recon/profiles-profiles/profiler
[*] Module installed: recon/profiles-profiles/twitter_mentioned
[*] Module installed: recon/profiles-profiles/twitter_mentions
[*] Module installed: recon/profiles-repositories/github_repos
[*] Module installed: recon/repositories-profiles/github_commits
[*] Module installed: recon/repositories-vulnerabilities/gists_search
[*] Module installed: recon/repositories-vulnerabilities/github_dorks
[*] Module installed: reporting/csv
[*] Module installed: reporting/html
[*] Module installed: reporting/json
[*] Module installed: reporting/list
[*] Module installed: reporting/proxifier
[*] Module installed: reporting/pushpin
[*] Module installed: reporting/xlsx
[*] Module installed: reporting/xml
[*] Reloading modules...
[!] 'google_api' key not set. pushpin module will likely fail at runtime. See 'keys add'.
[!] 'google_api' key not set. geocode module will likely fail at runtime. See 'keys add'.
[!] 'google_api' key not set. reverse_geocode module will likely fail at runtime. See 'keys add'.
[!] 'virustotal_api' key not set. virustotal module will likely fail at runtime. See 'keys add'.
[!] 'ipinfodb_api' key not set. ipinfodb module will likely fail at runtime. See 'keys add'.
[!] 'ipstack_api' key not set. ipstack module will likely fail at runtime. See 'keys add'.
[!] 'bing_api' key not set. bing_ip module will likely fail at runtime. See 'keys add'.
[!] 'shodan_api' key not set. shodan_org module will likely fail at runtime. See 'keys add'.
[!] 'github_api' key not set. github_miner module will likely fail at runtime. See 'keys add'.
[!] Module 'recon/domains-contacts/metacrawler' disabled. Dependency required: ''PyPDF3''.
[!] 'namechk_api' key not set. namechk module will likely fail at runtime. See 'keys add'.
[!] 'twitter_api' key not set. twitter_mentioned module will likely fail at runtime. See 'keys add'.
[!] 'twitter_secret' key not set. twitter_mentioned module will likely fail at runtime. See 'keys add'.
[!] 'twitter_api' key not set. twitter_mentions module will likely fail at runtime. See 'keys add'.
[!] 'twitter_secret' key not set. twitter_mentions module will likely fail at runtime. See 'keys add'.
[!] 'censysio_id' key not set. censysio module will likely fail at runtime. See 'keys add'.
[!] 'censysio_secret' key not set. censysio module will likely fail at runtime. See 'keys add'.
[!] 'pwnedlist_api' key not set. domain_ispwned module will likely fail at runtime. See 'keys add'.
[!] 'pwnedlist_secret' key not set. domain_ispwned module will likely fail at runtime. See 'keys add'.
[!] Module 'recon/domains-credentials/pwnedlist/account_creds' disabled. Dependency required: ''pyaes''.
[!] 'pwnedlist_api' key not set. leaks_dump module will likely fail at runtime. See 'keys add'.
[!] 'pwnedlist_secret' key not set. leaks_dump module will likely fail at runtime. See 'keys add'.
[!] Module 'recon/domains-credentials/pwnedlist/domain_creds' disabled. Dependency required: ''pyaes''.
[!] 'pwnedlist_api' key not set. api_usage module will likely fail at runtime. See 'keys add'.
[!] 'pwnedlist_secret' key not set. api_usage module will likely fail at runtime. See 'keys add'.
[!] 'shodan_api' key not set. shodan_ip module will likely fail at runtime. See 'keys add'.
[!] 'binaryedge_api' key not set. binaryedge module will likely fail at runtime. See 'keys add'.
[!] 'twitter_api' key not set. twitter module will likely fail at runtime. See 'keys add'.
[!] 'twitter_secret' key not set. twitter module will likely fail at runtime. See 'keys add'.
[!] 'shodan_api' key not set. shodan module will likely fail at runtime. See 'keys add'.
[!] 'google_api' key not set. youtube module will likely fail at runtime. See 'keys add'.
[!] 'flickr_api' key not set. flickr module will likely fail at runtime. See 'keys add'.
[!] 'shodan_api' key not set. shodan_net module will likely fail at runtime. See 'keys add'.
[!] 'virustotal_api' key not set. virustotal module will likely fail at runtime. See 'keys add'.
[!] 'hibp_api' key not set. hibp_breach module will likely fail at runtime. See 'keys add'.
[!] 'hibp_api' key not set. hibp_paste module will likely fail at runtime. See 'keys add'.
[!] 'github_api' key not set. github_dorks module will likely fail at runtime. See 'keys add'.
[!] 'hashes_api' key not set. hashes_org module will likely fail at runtime. See 'keys add'.
[!] 'fullcontact_api' key not set. fullcontact module will likely fail at runtime. See 'keys add'.
[!] Module 'recon/contacts-contacts/abc' disabled. Dependency required: ''bs4''.
[!] 'bing_api' key not set. bing_linkedin_cache module will likely fail at runtime. See 'keys add'.
[!] 'github_api' key not set. github_repos module will likely fail at runtime. See 'keys add'.
[!] 'github_api' key not set. github_commits module will likely fail at runtime. See 'keys add'.
[!] 'bing_api' key not set. bing_linkedin_contacts module will likely fail at runtime. See 'keys add'.
[!] 'github_api' key not set. github_users module will likely fail at runtime. See 'keys add'.
[!] 'builtwith_api' key not set. builtwith module will likely fail at runtime. See 'keys add'.
[!] 'bing_api' key not set. bing_domain_api module will likely fail at runtime. See 'keys add'.
[!] 'shodan_api' key not set. shodan_hostname module will likely fail at runtime. See 'keys add'.
[!] 'binaryedge_api' key not set. binaryedge module will likely fail at runtime. See 'keys add'.
[recon-ng][default] >
lanmaster53 commented 4 years ago

Even so, what you are experiencing is completely expected. Please read the wiki link from above. It is up to the user to install a module's dependencies prior to installing the module. If you install every module in the marketplace without considering any of their dependencies, then yes, you are going to get errors.

BBhacKing commented 4 years ago

"...completely expected" - it's not what I expected, haha. :-)

I will take a step back and try to adjust my assumptions. Thanks for the feedback.

lanmaster53 commented 4 years ago

Well... completely expected if you read the wiki. ;-)