search

elceef / dnstwist

Domain name permutation engine for detecting homograph phishing attacks, typo squatting, and brand impersonation
https://dnstwist.it
Apache License 2.0
4.81k stars 764 forks source link

UnicodeEncodeError: 'latin-1' codec can't encode character #118

Closed prisas closed 3 years ago

prisas commented 3 years ago

I am having the error "UnicodeEncodeError: 'latin-1' codec can't encode character" after several weeks of using dnstwist without error and just updating it to the latest release version.

Below is an example:

$ dnstwist -r -m example.com

| |_ _ | | ()| | / ` | ' \/ | \ \ /\ / / / | | | (| | | | _ \ | \ V V /| _ \ | _,|| ||/_| _/_/ ||/__| {20201022}

Processing 1650 permutations ▒▒▒▒▒30%▒52%▒77%▒▒▒▒98%▒▒▒ 387 hits

Traceback (most recent call last): File "/usr/bin/dnstwist", line 33, in sys.exit(load_entry_point('dnstwist==20201022', 'console_scripts', 'dnstwist')()) File "/usr/lib/python3/dist-packages/dnstwist.py", line 956, in main print(create_cli(domains)) UnicodeEncodeError: 'latin-1' codec can't encode character '\u0229' in position 3628: ordinal not in range(256)

I have the same problem if I go to the site-packages folder and execute the script explicitly with python 3.8.5. I installed the latest version using apt.

What could be causing this error?

elceef commented 3 years ago

What's your terminal encoding?

python3 -c "import sys; print(sys.stdout.encoding)"
prisas commented 3 years ago

It says it is the following: iso8859-1

elceef commented 3 years ago

Could you make sure your terminal encoding is set to UTF-8 and then check again? Is this Kali Linux?

prisas commented 3 years ago

I have set the terminal to UTF-8 modifying the .bashrc file with:

export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 export LANGUAGE=en_US.UTF-8

And now dnstwist is working again but it is reporting a lot of new domains that were not reported before, mostly with strange characters:

image

Is there a way to not display these new results? Or is this a new feature in the latest update?

Yes, it is a Kali Linux 2020.4 machine.

elceef commented 3 years ago

What's your DNS resolver configuration? Try running with --nameservers 8.8.8.8.

prisas commented 3 years ago

That seems to solve it! Thanks @elceef

I ran the command as: dnstwist -r example-com --nameserver 8.8.8.8

And it returned the following output:

Processing 1650 permutations ···21%·51%·83%·· 98 hits

original* example.com 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 NS:a.iana-servers.net MX: addition examplea.com 3.223.115.185 NS:nsg1.namebrightdns.com addition exampleb.com 3.223.115.185 NS:ns1.namebrightdns.com addition examplec.com 3.223.115.185 NS:nsg1.namebrightdns.com addition exampled.com 91.195.241.137 NS:ns1.sedoparking.com MX:localhost addition examplee.com 167.88.193.189 NS:jm1.dns.com addition exampleg.com 202.181.185.161 NS:ns6.timway.com MX:mx.zoho.com addition examplei.com 3.134.22.63 2600:1f16:389:3100:788f:e5de:ee5b:1161 addition examplej.com 156.243.243.187 NS:f1g1ns1.dnspod.net addition examplel.com 138.201.138.240 2a01:4f8:172:35ec::2 NS:b.ns14.net MX:mail.examplel.com addition examplen.com 156.243.248.170 NS:f1g1ns1.dnspod.net addition exampleq.com 192.241.217.167 NS:ns1.slicehost.net addition exampler.com 66.111.4.53 NS:ns1.messagingengine.com MX:in1-smtp.messagingengine.com addition examples.com 3.211.72.104 NS:ns-1064.awsdns-05.org MX:alt1.aspmx.l.google.com addition examplet.com 34.102.136.180 NS:ns23.domaincontrol.com MX:mailstore1.secureserver.net addition exampleu.com !ServFail NS:sk.s5.ans1.ns148.ztomy.com addition examplew.com 52.0.217.44 NS:ns1.dynadot.com addition examplex.com 52.128.23.153 NS:ns1.uniregistrymarket.link MX:localhost addition exampley.com NS:capricorn.dnspod.net addition examplez.com 52.128.23.153 NS:ns1.uniregistrymarket.link MX:localhost bitsquatting axample.com 204.11.56.48 NS:ns1626.ztomy.com bitsquatting uxample.com 160.124.76.5 NS:ns4.dnsdun.com bitsquatting ezample.com 3.223.115.185 NS:nsg1.namebrightdns.com bitsquatting ehample.com 156.238.56.66 NS:ns6.dnsdun.com bitsquatting excmple.com 45.207.166.217 NS:ns51.domaincontrol.com bitsquatting exemple.com 107.180.40.145 NS:ns49.domaincontrol.com bitsquatting eximple.com 184.168.131.241 NS:ns73.domaincontrol.com bitsquatting exqmple.com NS:ns-1082.awsdns-07.org bitsquatting exalple.com 81.17.18.198 NS:ns1.panamans.com MX:mail.h-email.net bitsquatting examqle.com 74.208.236.54 2607:f1c0:100f:f000::2ee NS:ns1063.ui-dns.de MX:examqle-com.mail.protection.outlook.com bitsquatting exampld.com 185.53.179.29 NS:ns1mtw.name.com bitsquatting exampla.com 52.128.23.153 NS:ns1.uniregistrymarket.link MX:localhost bitsquatting examplu.com 157.7.156.160 NS:dns01.gmoserver.jp MX:mx11.gmoserver.jp homoglyph exämple.com 85.13.149.201 NS:ns5.kasserver.com MX:w010823c.kasserver.com homoglyph cxample.com 104.217.244.220 NS:jm1.dns.com homoglyph examplc.com 108.186.54.30 NS:ns1.dnsowl.com homoglyph éxample.com 45.79.105.33 2600:3c01::f03c:91ff:fe37:1c9f NS:q.secondary2.com homoglyph exarnple.com 217.70.184.38 NS:ns-122-c.gandi.net MX:fb.mail.gandi.net homoglyph exampie.com 208.91.196.40 NS:sk.s5.ans1.ns116.ztomy.com homoglyph exanp1e.com 104.31.68.59 2606:4700:3034::681f:443b NS:nola.ns.cloudflare.com MX:dc-9fe311c79b55.exanp1e.com homoglyph exanmple.com 91.195.240.94 NS:ns1hnx.name.com homoglyph exammple.com 3.223.115.185 NS:nsg1.namebrightdns.com homoglyph examp1e.com 64.57.183.2 2001:470:1f07:1126::555:1212 NS:osdn.iecc.com MX:mx2.examp1e.com homoglyph cxamplc.com 142.252.183.45 NS:ns1.dns.com homoglyph exanple.com 88.214.207.96 NS:ns3.epik.com MX:mx37.m1bp.com homoglyph exannple.com NS:ns-1030.awsdns-00.org homoglyph exampic.com 104.31.78.41 2606:4700:3032::681f:4e29 NS:rick.ns.cloudflare.com MX:eforward1.registrar-servers.com homoglyph exarmple.com 109.70.26.37 NS:ns3.nic.ru hyphenation ex-ample.com 67.227.226.240 NS:ns1.parklogic.com MX:mx156.hostedmxserver.com hyphenation exa-mple.com 91.195.241.137 NS:ns1.sedoparking.com MX:localhost hyphenation exam-ple.com 72.52.179.174 NS:ns1.parklogic.com MX:mail.hope-mail.com hyphenation examp-le.com 72.52.179.174 NS:ns1.parklogic.com MX:mail.hope-mail.com insertion exampple.com 107.180.50.237 NS:ns19.domaincontrol.com MX:exampple.com insertion exapmple.com 39.109.116.74 NS:dns13.hichina.com insertion excample.com 81.171.22.4 NS:ns1.weaponizedcow.com insertion examlple.com 163.197.226.21 NS:jm1.dns.com insertion exsample.com 95.211.189.137 NS:ns1.smtmdns.com insertion examople.com 165.3.106.129 NS:ns1.dns.com insertion examplle.com 104.28.20.101 2606:4700:3032::681c:1565 NS:eric.ns.cloudflare.com insertion exzample.com 107.186.71.9 NS:jm1.dns.com omission xample.com 78.41.204.32 NS:ns1.torresdns.com omission exmple.com 67.210.233.131 NS:ns19.globalcon.net MX:localhost.exmple.com omission examle.com 104.200.22.130 NS:ns49.domaincontrol.com omission eample.com 52.128.23.153 NS:ns1.uniregistrymarket.link MX:localhost omission exaple.com 99.83.191.119 NS:ns1.parkingcrew.net MX:mail.h-email.net omission exampl.com 198.58.118.167 NS:ns1.mytrafficmanagement.com omission exampe.com 72.52.179.174 NS:ns1.parklogic.com MX:mx156.hostedmxserver.com repetition eexample.com 96.38.240.54 NS:ns1.onlineearth.com MX:mail.onlineearth.com repetition exaample.com !ServFail !ServFail NS:!ServFail repetition exxample.com 34.102.136.180 NS:ns69.domaincontrol.com MX:mailstore1.secureserver.net replacement exampl3.com NS:ns-cloud-a1.googledomains.com replacement examole.com 72.52.179.174 NS:ns1.parklogic.com MX:mx156.hostedmxserver.com replacement exampls.com 23.225.230.169 NS:juming.dnsdun.com replacement ecample.com 23.92.78.154 NS:ns1.dnsowl.com MX:mx.zoho.eu replacement 4xample.com 3.223.115.185 NS:nsg1.namebrightdns.com replacement exzmple.com 154.93.217.77 NS:ns1.dns.com replacement exsmple.com 3.223.115.185 NS:nsg1.namebrightdns.com replacement examplw.com 103.224.182.246 NS:ns1.above.com MX:park-mx.above.com replacement rxample.com 103.224.182.246 NS:ns1.above.com MX:park-mx.above.com replacement exapple.com 208.91.197.27 NS:ns95.worldnic.com replacement examplr.com 204.11.56.48 NS:ns1626.ztomy.com replacement esample.com !ServFail !ServFail NS:!ServFail replacement wxample.com 72.52.179.174 NS:ns1.parklogic.com MX:mx156.hostedmxserver.com replacement sxample.com 185.53.177.51 NS:ns1.parkingcrew.net MX:mail.h-email.net replacement exwmple.com 199.59.242.153 NS:ns1.bodis.com MX:mx76.m2bp.com subdomain e.xample.com 78.41.204.32 subdomain exa.mple.com !ServFail !ServFail NS:!ServFail transposition xeample.com NS:delta.netnautics.net MX:mx.xeample.com transposition eaxmple.com 157.7.236.66 2001:e42:102:1101:202:181:103:212 transposition exmaple.com 104.18.144.11 2606:4700::6812:900b NS:ara.ns.cloudflare.com transposition exapmle.com 103.224.182.207 NS:ns1.above.com MX:park-mx.above.com transposition examlpe.com 173.239.5.6 NS:ns1.expiereddnsmanager.com MX:mx7.examlpe.com transposition exampel.com 72.52.179.175 NS:ns1.parklogic.com MX:mx174.mb5p.com vowel-swap examplo.com 3.223.115.185 NS:nsg1.namebrightdns.com vowel-swap exampli.com 216.239.32.21 2001:4860:4802:32::15 NS:ns-cloud-e1.googledomains.com vowel-swap ixample.com 127.0.0.1 NS:nsv0.nitrado.net MX:ixample.com vowel-swap exomple.com 104.27.158.123 2606:4700:3031::681b:9f7b NS:connie.ns.cloudflare.com MX:mx-biz.mail.am0.yahoodns.net various examplecom.com 3.223.115.185 NS:nsg1.namebrightdns.com