search

Homebrew / homebrew-core

🍻 Default formulae for the missing package manager for macOS (or Linux)
https://brew.sh
BSD 2-Clause "Simplified" License
13.81k stars 12.48k forks source link

[Formula/dnstwist] Wrong Python version used, resulting in broken command #26144

Closed wichert closed 6 years ago

wichert commented 6 years ago

The dnstwist command is installed to use the standard python binary, which means Python 2.7:

#!/bin/bash
PYTHONPATH="/usr/local/Cellar/dnstwist/1.02/libexec/vendor/lib/python2.7/site-packages" exec "/usr/local/Cellar/dnstwist/1.02/libexec/bin/dnstwist" "$@"

However dnstwisted uses the iteritems() method for a map, which was only added in Python 3. This results in the dnstwist command never working:

$ dnstwist 
Traceback (most recent call last):
  File "/usr/local/Cellar/dnstwist/1.02/libexec/bin/dnstwist", line 42, in <module>
    import dns.resolver
  File "/usr/local/Cellar/dnstwist/1.02/libexec/vendor/lib/python2.7/site-packages/dns/resolver.py", line 32, in <module>
    import dns.flags
  File "/usr/local/Cellar/dnstwist/1.02/libexec/vendor/lib/python2.7/site-packages/dns/flags.py", line 51, in <module>
    _by_value = dict([(y, x) for x, y in _by_text.iteritems()])
AttributeError: 'dict' object has no attribute 'iteritems'
ilovezfs commented 6 years ago

It seems to be fine to me

iMac-TMP:~ joe$ brew instal dnstwist
==> Installing dependencies for dnstwist: ssdeep, geoip
==> Installing dnstwist dependency: ssdeep
==> Downloading https://homebrew.bintray.com/bottles/ssdeep-2.14.1.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring ssdeep-2.14.1.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/ssdeep/2.14.1: 15 files, 144.8KB
==> Installing dnstwist dependency: geoip
==> Downloading https://homebrew.bintray.com/bottles/geoip-1.6.12.el_capitan.bottle.tar.gz
Already downloaded: /Users/joe/Library/Caches/Homebrew/geoip-1.6.12.el_capitan.bottle.tar.gz
==> Pouring geoip-1.6.12.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/geoip/1.6.12: 18 files, 548.7KB
==> Installing dnstwist
==> Downloading https://homebrew.bintray.com/bottles/dnstwist-1.02.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring dnstwist-1.02.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/dnstwist/1.02: 250 files, 3.7MB
iMac-TMP:~ joe$ dnstwist
dnstwist 1.02 by <marcin@ulikowski.pl>

usage: /usr/local/Cellar/dnstwist/1.02/libexec/bin/dnstwist [OPTION]... DOMAIN

Find similar-looking domain names that adversaries can use to attack you. Can
detect typosquatters, phishing attacks, fraud and corporate espionage. Useful
as an additional source of targeted threat intelligence.

positional arguments:
  domain                domain name or URL to check

optional arguments:
  -h, --help            show this help message and exit
  -c, --csv             print output in CSV format
  -j, --json            print output in JSON format
  -r, --registered      show only registered domain names
  -w, --whois           perform lookup for WHOIS creation/update time (slow)
  -g, --geoip           perform lookup for GeoIP location
  -b, --banners         determine HTTP and SMTP service banners
  -s, --ssdeep          fetch web pages and compare their fuzzy hashes to
                        evaluate similarity
  -m, --mxcheck         check if MX host can be used to intercept e-mails
  -d FILE, --dictionary FILE
                        generate additional domains using dictionary FILE
  -t NUMBER, --threads NUMBER
                        start specified NUMBER of threads (default: 10)
iMac-TMP:~ joe$ dnstwist example.com
     _           _            _     _   
  __| |_ __  ___| |___      _(_)___| |_ 
 / _` | '_ \/ __| __\ \ /\ / / / __| __|
| (_| | | | \__ \ |_ \ V  V /| \__ \ |_ 
 \__,_|_| |_|___/\__| \_/\_/ |_|___/\__| {1.02}

Processing 171 domain variants ..33%.69%..96% 73 hits (42%)

Original*       example.com       93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 NS:sns.dns.icann.org
Addition        examplea.com      52.86.22.136 NS:ns1.namebrightdns.com
Addition        exampleb.com      52.86.22.136 NS:ns1.namebrightdns.com
Addition        examplec.com      52.86.22.136 NS:ns1.namebrightdns.com
Addition        exampled.com      72.52.4.119 NS:ns1.sedoparking.com MX:localhost
Addition        examplee.com      -
Addition        examplef.com      -
Addition        exampleg.com      202.181.185.161 NS:ns6.timway.com MX:mx.zoho.com
Addition        exampleh.com      185.53.177.31 NS:ns1vwx.name.com
Addition        examplei.com      184.172.31.83 NS:ns1.rrpproxy.net
Addition        examplej.com      -
Addition        examplek.com      -
Addition        examplel.com      46.4.96.137 2a01:4f8:130:2303::2 NS:ns.examplel.com MX:mail.examplel.com
Addition        examplem.com      -
Addition        examplen.com      -
Addition        exampleo.com      -
Addition        examplep.com      -
Addition        exampleq.com      67.23.3.156 NS:ns1.slicehost.net
Addition        exampler.com      66.111.4.53 NS:ns1.messagingengine.com MX:in1-smtp.messagingengine.com
Addition        examples.com      18.233.121.245 NS:ns-227.awsdns-28.com MX:aspmx.l.google.com
Addition        examplet.com      50.63.202.48 NS:ns23.domaincontrol.com MX:smtp.secureserver.net
Addition        exampleu.com      185.27.134.162 NS:ns2.exampleu.com MX:mx.exampleu.com
Addition        examplev.com      -
Addition        examplew.com      -
Addition        examplex.com      37.58.107.93 NS:ns1.eftydns.com MX:mail.efty.com
Addition        exampley.com      108.167.143.133 NS:ns8351.hostgator.com MX:mail.exampley.com
Addition        examplez.com      69.172.201.153 NS:ns1.uniregistrymarket.link
Bitsquatting    dxample.com       66.34.73.27 NS:ns1.7artisan.cloud MX:dxample.com
Bitsquatting    gxample.com       -
Bitsquatting    axample.com       204.11.56.48 NS:ns1626.ztomy.com
Bitsquatting    mxample.com       -
Bitsquatting    uxample.com       66.6.44.4 NS:ns71.domaincontrol.com MX:smtp.secureserver.net
Bitsquatting    eyample.com       -
Bitsquatting    ezample.com       204.11.56.48 NS:ns1626.ztomy.com
Bitsquatting    epample.com       -
Bitsquatting    ehample.com       185.53.177.31 NS:ns1vwx.name.com
Bitsquatting    e8ample.com       -
Bitsquatting    excmple.com       -
Bitsquatting    exemple.com       68.178.213.61 NS:ns1.namefind.com
Bitsquatting    eximple.com       206.188.192.139 NS:ns93.worldnic.com MX:p.webcom.ctmail.com
Bitsquatting    exqmple.com       -
Bitsquatting    exalple.com       -
Bitsquatting    exaople.com       -
Bitsquatting    exaiple.com       -
Bitsquatting    exaeple.com       -
Bitsquatting    exa-ple.com       -
Bitsquatting    examqle.com       162.255.119.249 NS:dns1.registrar-servers.com MX:eforward1.registrar-servers.com
Bitsquatting    examrle.com       -
Bitsquatting    examtle.com       -
Bitsquatting    examxle.com       -
Bitsquatting    exam0le.com       -
Bitsquatting    exampme.com       -
Bitsquatting    exampne.com       184.172.106.42 NS:ns1.milesmx.com
Bitsquatting    examphe.com       184.168.221.34 NS:ns09.domaincontrol.com
Bitsquatting    exampde.com       -
Bitsquatting    exampld.com       185.53.179.29 NS:ns1mtw.name.com
Bitsquatting    examplg.com       -
Bitsquatting    exampla.com       69.172.201.153 NS:ns1.uniregistrymarket.link
Bitsquatting    examplm.com       -
Bitsquatting    examplu.com       157.7.156.160 NS:dns01.gmoserver.jp MX:mx11.gmoserver.jp
Homoglyph       exarrple.com      -
Homoglyph       exanple.com       192.64.147.140 NS:ns3.epik.com MX:mx37.mb5p.com
Homoglyph       exannple.com      -
Homoglyph       exampie.com       208.91.196.40 NS:sk.s5.ans1.ns116.ztomy.com
Homoglyph       exarnple.com      -
Homoglyph       examp1e.com       64.57.183.2 2001:470:1f07:1126::555:1212 NS:sdn.iecc.com MX:mx.examp1e.com
Hyphenation     e-xample.com      95.142.10.206 NS:ns.viatecla.pt MX:aspmx.l.google.com
Hyphenation     ex-ample.com      -
Hyphenation     exa-mple.com      -
Hyphenation     exam-ple.com      -
Hyphenation     examp-le.com      -
Hyphenation     exampl-e.com      -
Insertion       exakmple.com      -
Insertion       exammple.com      81.177.135.182 NS:ns1.jino.ru
Insertion       exasmple.com      185.53.177.31 NS:ns1dns.name.com
Insertion       examjple.com      -
Insertion       exajmple.com      -
Insertion       exawmple.com      -
Insertion       exaqmple.com      -
Insertion       exanmple.com      -
Insertion       exwample.com      -
Insertion       ex2ample.com      -
Insertion       exa2mple.com      -
Insertion       exaymple.com      -
Insertion       eyxample.com      -
Insertion       exzample.com      192.64.147.150 NS:ns1.dynadot.com MX:mx37.m1bp.com
Insertion       exapmple.com      -
Insertion       ezxample.com      -
Insertion       exam0ple.com      -
Insertion       exampole.com      184.168.221.50 NS:ns09.domaincontrol.com
Insertion       examploe.com      -
Insertion       ex1ample.com      -
Insertion       examplke.com      -
Insertion       excample.com      67.55.92.184 NS:excample.com MX:mx7.excample.com
Insertion       exalmple.com      -
Insertion       ecxample.com      -
Insertion       examplme.com      -
Insertion       exazmple.com      -
Insertion       examople.com      -
Insertion       exampkle.com      -
Insertion       exsample.com      216.157.88.22 NS:ns1.smtmdns.com
Insertion       esxample.com      -
Insertion       edxample.com      -
Insertion       exa1mple.com      -
Insertion       exdample.com      -
Insertion       examp0le.com      -
Insertion       exyample.com      -
Insertion       examplpe.com      -
Insertion       exampple.com      192.186.210.35 NS:ns19.domaincontrol.com MX:exampple.com
Insertion       examlple.com      -
Insertion       exqample.com      -
Insertion       examkple.com      -
Insertion       examplle.com      91.134.15.69 NS:ns.examplle.com MX:mail.examplle.com
Insertion       examnple.com      -
Insertion       exampmle.com      -
Omission        eample.com        69.172.201.153 NS:ns1.uniregistrymarket.link
Omission        exaple.com        67.55.92.182 NS:exaple.com MX:mx7.exaple.com
Omission        xample.com        209.126.123.13 NS:ns1.torresdns.com
Omission        examle.com        68.178.213.61 NS:ns1.namefind.com
Omission        exampl.com        141.8.224.169 NS:ns27.rookdns.com
Omission        exampe.com        67.225.218.50 NS:ns1.parklogic.com
Omission        exmple.com        67.210.233.131 NS:ns19.globalcon.net MX:localhost.exmple.com
Repetition      eexample.com      96.38.240.54 NS:ns1.onlineearth.com MX:mail.onlineearth.com
Repetition      exxample.com      50.63.202.51 NS:ns69.domaincontrol.com MX:smtp.secureserver.net
Repetition      exaample.com      52.27.119.147 NS:ns-187.awsdns-23.com
Replacement     exampoe.com       -
Replacement     examlle.com       -
Replacement     exampl3.com       NS:ns-cloud-a1.googledomains.com MX:gmr-smtp-in.l.google.com
Replacement     exammle.com       -
Replacement     wxample.com       67.225.218.50 NS:ns1.parklogic.com
Replacement     examppe.com       -
Replacement     examole.com       67.225.218.50 NS:ns1.parklogic.com
Replacement     ex2mple.com       -
Replacement     3xample.com       198.54.117.200 NS:dns101.registrar-servers.com
Replacement     exapple.com       208.91.197.27 NS:ns95.worldnic.com MX:p.webcom.ctmail.com
Replacement     exzmple.com       67.222.8.138 NS:ns1.canetree.com MX:exzmple.com
Replacement     ecample.com       -
Replacement     exakple.com       -
Replacement     exampl4.com       -
Replacement     ex1mple.com       -
Replacement     exwmple.com       199.59.242.150 NS:ns1dns.name.com
Replacement     rxample.com       204.11.56.48 NS:ns1626.ztomy.com
Replacement     exampke.com       -
Replacement     exymple.com       185.53.177.31 NS:ns1dns.name.com
Replacement     4xample.com       52.86.22.136 NS:ns1.namebrightdns.com
Replacement     examplr.com       -
Replacement     exsmple.com       52.86.22.136 NS:ns1.namebrightdns.com
Replacement     sxample.com       104.200.22.130 NS:localhost MX:mail.mxproc.com
Replacement     examplz.com       -
Replacement     zxample.com       -
Replacement     exampls.com       -
Replacement     examplw.com       -
Replacement     esample.com       -
Replacement     edample.com       -
Replacement     exajple.com       -
Subdomain       e.xample.com      -
Subdomain       ex.ample.com      -
Subdomain       exa.mple.com      -
Subdomain       exam.ple.com      -
Subdomain       examp.le.com      -
Subdomain       exampl.e.com      -
Transposition   xeample.com       NS:gamma.netnautics.net MX:mx.xeample.com
Transposition   eaxmple.com       157.7.236.66 2001:e42:102:1101:202:181:103:212 NS:dns1.eaxmple.com
Transposition   exmaple.com       185.73.44.35 2001:ba8:0:2c23:: NS:a.authns.bitfolk.co.uk MX:numpty.absolutelyplastered.com
Transposition   exapmle.com       103.224.182.207 NS:ns1.above.com MX:park-mx.above.com
Transposition   examlpe.com       173.239.5.6 NS:examlpe.com MX:mx7.examlpe.com
Transposition   exampel.com       67.227.226.241 NS:ns1.parklogic.com MX:mx174.mb5p.com
Various         wwexample.com     67.55.92.184 NS:wwexample.com MX:mx7.wwexample.com
Various         wwwexample.com    208.73.210.202 NS:ns1.dsredirection.com
Various         www-example.com   209.217.249.112 NS:ns1.mymanagedvps.com MX:www-example.com
Various         examplecom.com    69.172.201.153 NS:ns1.uniregistrymarket.link
iMac-TMP:~ joe$ brew test -vd dnstwist
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/dnstwist.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/ssdeep.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/geoip.rb
Testing dnstwist
/usr/bin/sandbox-exec -f /tmp/homebrew20180403-50867-12dzjvn.sb /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.3/bin/ruby -W0 -I /usr/local/Homebrew/Library/Homebrew -- /usr/local/Homebrew/Library/Homebrew/test.rb /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/dnstwist.rb -vd
/usr/local/Homebrew/Library/Homebrew/test.rb (Formulary::FromPathLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/dnstwist.rb
==> /usr/local/Cellar/dnstwist/1.02/bin/dnstwist github.com
iMac-TMP:~ joe$
wichert commented 6 years ago

@ilovezfs That's interesting. Is your python a Python 3, or is /usr/local/bin/dnstwist differently for you?

wichert commented 6 years ago

One possibly relevant difference is that I am using high sierra, resulting in different bottles being used:

==> Installing dependencies for dnstwist: ssdeep, geoip
==> Installing dnstwist dependency: ssdeep
==> Downloading https://homebrew.bintray.com/bottles/ssdeep-2.14.1.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring ssdeep-2.14.1.high_sierra.bottle.tar.gz
🍺  /usr/local/Cellar/ssdeep/2.14.1: 15 files, 144.9KB
==> Installing dnstwist dependency: geoip
==> Downloading https://homebrew.bintray.com/bottles/geoip-1.6.12.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring geoip-1.6.12.high_sierra.bottle.tar.gz
🍺  /usr/local/Cellar/geoip/1.6.12: 18 files, 548.9KB
==> Installing dnstwist
==> Downloading https://homebrew.bintray.com/bottles/dnstwist-1.02.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring dnstwist-1.02.high_sierra.bottle.tar.gz
🍺  /usr/local/Cellar/dnstwist/1.02: 251 files, 3.7MB
ilovezfs commented 6 years ago 
iMac-TMP:~ joe$ which python
/usr/local/bin/python
iMac-TMP:~ joe$ /usr/local/bin/python --version
Python 2.7.14
iMac-TMP:~ joe$ mdutil -t /usr/local/bin/python
/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/bin/python2.7
iMac-TMP:~ joe$ ls -l /usr/local/bin/python
lrwxr-xr-x  1 joe  staff  38 Apr  3 20:50 /usr/local/bin/python -> ../Cellar/python@2/2.7.14_3/bin/python
iMac-TMP:~ joe$
wichert commented 6 years ago

@ilovezfs How about the contents of /usr/local/bin/dnstwist ? Which Python version does that use?

ilovezfs commented 6 years ago

The error seems to occur if your python is python3.

ilovezfs commented 6 years ago

Note

iMac-TMP:~ joe$ cat /usr/local/bin/dnstwist
#!/bin/bash
PYTHONPATH="/usr/local/Cellar/dnstwist/1.02/libexec/vendor/lib/python2.7/site-packages" exec "/usr/local/Cellar/dnstwist/1.02/libexec/bin/dnstwist" "$@"
iMac-TMP:~ joe$ head -1 /usr/local/Cellar/dnstwist/1.02/libexec/bin/dnstwist
#!/usr/bin/env python
iMac-TMP:~ joe$
ilovezfs commented 6 years ago 
iMac-TMP:~ joe$ export PATH=/usr/local/opt/python/libexec/bin:$PATH
iMac-TMP:~ joe$ dnstwist
Traceback (most recent call last):
  File "/usr/local/Cellar/dnstwist/1.02/libexec/bin/dnstwist", line 42, in <module>
    import dns.resolver
  File "/usr/local/Cellar/dnstwist/1.02/libexec/vendor/lib/python2.7/site-packages/dns/resolver.py", line 32, in <module>
    import dns.flags
  File "/usr/local/Cellar/dnstwist/1.02/libexec/vendor/lib/python2.7/site-packages/dns/flags.py", line 51, in <module>
    _by_value = dict([(y, x) for x, y in _by_text.iteritems()])
AttributeError: 'dict' object has no attribute 'iteritems'
wichert commented 6 years ago

Ah, you're right. Removing /usr/local/opt/python/libexec/bin from PATH does fix this.

To make this more robust perhaps it makes sense to modify /usr/local/Cellar/dnstwist/1.02/libexec/bin/dnstwist to explicitly refer to python3 instead of python.

ilovezfs commented 6 years ago

I think you mean to explicitly refer to python 2. It breaks with python 3.

wichert commented 6 years ago

You're right of course!

ilovezfs commented 6 years ago

In general we don't yet consider the use of #!/usr/bin/env python for Python 2 to be a bug due to https://www.python.org/dev/peps/pep-0394/ but this may be something that we start auditing out of existence eventually.