search

owasp-amass / amass

In-depth attack surface mapping and asset discovery
https://owasp.org/www-project-amass/
Other
11.63k stars 1.85k forks source link

"No names were discovered" even though hostnames do resolve #349

Closed djcater closed 4 years ago

djcater commented 4 years ago

I am filing this as a new issue to avoid clogging up a closed issue (#305).

I am seeing this with the latest 3.4.2 using the release binary for Linux x64 on Ubuntu 19.10.

-passive will return hundreds of results, but replacing -passive with -ip will return no (or occasionally very few) results, even though spot-checking some of the subdomains found passively shows that they do actually resolve to IP addresses (or CNAMEs).

Here's a minimal example which demonstrates the above:

$ ./amass enum -v -passive -src -noalts -include Pastebin -d twitter.com 
Querying Pastebin for twitter.com subdomains
[Pastebin]        pic.twitter.com
[Pastebin]        mobile.twitter.com
[Pastebin]        happening.mobile.twitter.com
[Pastebin]        careers.twitter.com
[Pastebin]        www.twitter.com
[Pastebin]        university.twitter.com
[Pastebin]        hoping.twitter.com
[Pastebin]        blog.twitter.com
[Pastebin]        about.twitter.com
[Pastebin]        search.twitter.com
[Pastebin]        support.twitter.com
[Pastebin]        developer.twitter.com
[Pastebin]        1pic.twitter.com
[Pastebin]        tpic.twitter.com
[Pastebin]        connected.twitter.com
[Pastebin]        tweetdeck.twitter.com
[Pastebin]        api.twitter.com
[Pastebin]        nwww.twitter.com
[Pastebin]        studio.twitter.com
[Pastebin]        ads-api.twitter.com

OWASP Amass v3.4.2                                https://github.com/OWASP/Amass
--------------------------------------------------------------------------------
20 names discovered - api: 20

Then clear the cache to remove any side effects:

$ rm -r ~/.config/amass/

And here's the same command but with -passive replaced with -ip:

$ ./amass enum -v -ip -src -noalts -include Pastebin -d twitter.com 
Querying Pastebin for twitter.com subdomains
No names were discovered

Spot checking one of the passive subdomains shows that it does in fact resolve:

$ host mobile.twitter.com 
mobile.twitter.com has address 104.244.42.6
mobile.twitter.com has address 104.244.42.198
mobile.twitter.com has address 104.244.42.134
mobile.twitter.com has address 104.244.42.70

This is on a home network with no DNS interception or rewriting, and no VPN (although I did try it over a VPN as well just to avoid any issues with my home network and got the same results).

I also built the Docker image and ran it that way, getting the same results ("No names were discovered").

(After some bisecting it seems to me like it stopped working between 3.0.3 and 3.0.4 but I'm not 100% sure).

If I capture the network traffic in Wireshark whilst the Amass command is running I can actually see the valid DNS response coming back from one of the DNS servers, but for some reason Amass is ignoring it (there are no messages printed about false-positives or false-negatives relating to the DNS resolvers):

image

@caffix: Please let me know if there's anything else I can do to help debug this? Thank you.

caffix commented 4 years ago

This is excellent feedback! Apologies for the delayed response. I have an idea as to what could be causing the issue. Until the exact reason is identified and a fix released, try setting a single DNS resolver using the ‘-r’ flag

I’m curious if setting the single resolver gives you different results

ngkogkos commented 4 years ago

@djcater awesome feedback! Can you also provide few lines extract of the amass log file for that example subdomain, if you still have it, as it may help? I am curious to see if it's same to what I was seeing once.

djcater commented 4 years ago

Thanks for the replies both.

I don't have the previous log, but I can still reproduce the issue so here is a new log:

17:37:03.548659 Querying Pastebin for twitter.com subdomains
17:37:11.548473 DNS: Resolver: tpic.twitter.com type CNAME returned 0 results
17:37:11.548529 DNS: Resolver: connected.twitter.com type CNAME returned 0 results
17:37:11.548609 DNS: Resolver: mobile.twitter.com type CNAME returned 0 results
17:37:11.548647 DNS: Resolver: mobile.twitter.com type TXT returned 0 results
17:37:11.548666 DNS: Resolver: tpic.twitter.com type TXT returned 0 results
17:37:11.548692 DNS: Resolver: developer.twitter.com type TXT returned 0 results
17:37:11.548718 DNS: Resolver: connected.twitter.com type TXT returned 0 results
17:37:11.548735 DNS: Resolver: pic.twitter.com type TXT returned 0 results
17:37:11.548751 DNS: Resolver: tpic.twitter.com type A returned 0 results
17:37:11.548768 DNS: Resolver: hoping.twitter.com type CNAME returned 0 results
17:37:11.548784 DNS: Resolver: mobile.twitter.com type AAAA returned 0 results
17:37:11.548892 DNS: Resolver: connected.twitter.com type A returned 0 results
17:37:11.548925 DNS: Resolver: blog.twitter.com type TXT returned 0 results
17:37:11.548944 DNS: Resolver: support.twitter.com type CNAME returned 0 results
17:37:11.548969 DNS: Resolver: 1pic.twitter.com type CNAME returned 0 results
17:37:11.548998 DNS: Resolver: connected.twitter.com type AAAA returned 0 results
17:37:11.549015 DNS: Resolver: hoping.twitter.com type TXT returned 0 results
17:37:11.549043 DNS: Resolver: search.twitter.com type TXT returned 0 results
17:37:11.549069 DNS: Resolver: tpic.twitter.com type AAAA returned 0 results
17:37:11.549085 DNS: Resolver: developer.twitter.com type AAAA returned 0 results
17:37:11.549104 DNS: Resolver: pic.twitter.com type AAAA returned 0 results
17:37:11.549138 DNS: Resolver: about.twitter.com type TXT returned 0 results
17:37:11.549164 DNS: Resolver: careers.twitter.com type TXT returned 0 results
17:37:11.549182 DNS: Resolver: university.twitter.com type TXT returned 0 results
17:37:11.549201 DNS: Resolver: support.twitter.com type TXT returned 0 results
17:37:11.549235 DNS: Resolver: 1pic.twitter.com type TXT returned 0 results
17:37:11.549259 DNS: Resolver: hoping.twitter.com type A returned 0 results
17:37:11.549276 DNS: Resolver: 1pic.twitter.com type A returned 0 results
17:37:11.549292 DNS: Resolver: about.twitter.com type AAAA returned 0 results
17:37:11.549327 DNS: Resolver: hoping.twitter.com type AAAA returned 0 results
17:37:11.549353 DNS: Resolver: university.twitter.com type AAAA returned 0 results
17:37:11.549370 DNS: Resolver: blog.twitter.com type AAAA returned 0 results
17:37:11.549386 DNS: Resolver: search.twitter.com type AAAA returned 0 results
17:37:11.549420 DNS: Resolver: 1pic.twitter.com type AAAA returned 0 results
17:37:11.549451 DNS: Resolver: support.twitter.com type AAAA returned 0 results
17:37:11.549483 DNS: Resolver: careers.twitter.com type AAAA returned 0 results
17:37:11.549514 DNS: Resolver: www.twitter.com type AAAA returned 0 results
17:37:11.549545 DNS: SPF record query error: twitter.com: Resolver: twitter.com type SPF returned 0 results
17:37:13.548556 DNS: Resolver: s.twitter.com type CNAME returned 0 results
17:37:13.548642 DNS: Resolver: s.twitter.com type TXT returned 0 results
17:37:13.548659 DNS: Resolver: s.twitter.com type AAAA returned 0 results
17:37:13.548680 DNS: Resolver: twitter.com type CNAME returned 0 results
17:37:13.548711 DNS: Resolver: alt2.aspmx.l.google.com type CNAME returned 0 results
17:37:13.548726 DNS: Resolver: d.r06.twtrdns.net type CNAME returned 0 results
17:37:13.548738 DNS: Resolver: alt1.aspmx.l.google.com type CNAME returned 0 results
17:37:13.548749 DNS: Resolver: a.r06.twtrdns.net type CNAME returned 0 results
17:37:13.548760 DNS: Resolver: ns3.p34.dynect.net type CNAME returned 0 results
17:37:13.548771 DNS: Resolver: b.r06.twtrdns.net type CNAME returned 0 results
17:37:13.548783 DNS: Resolver: c.r06.twtrdns.net type CNAME returned 0 results
17:37:13.548794 DNS: Resolver: aspmx.l.google.com type CNAME returned 0 results
17:37:13.548807 DNS: Resolver: aspmx2.googlemail.com type CNAME returned 0 results
17:37:13.548837 DNS: Resolver: d01-01.ns.twtrdns.net type CNAME returned 0 results
17:37:13.548849 DNS: Resolver: a.r06.twtrdns.net type TXT returned 0 results
17:37:13.548870 DNS: Resolver: ns4.p34.dynect.net type CNAME returned 0 results
17:37:13.548909 DNS: Resolver: ns1.p34.dynect.net type CNAME returned 0 results
17:37:13.548923 DNS: Resolver: d.r06.twtrdns.net type TXT returned 0 results
17:37:13.548935 DNS: Resolver: alt1.aspmx.l.google.com type TXT returned 0 results
17:37:13.548948 DNS: Resolver: aspmx3.googlemail.com type CNAME returned 0 results
17:37:13.548977 DNS: Resolver: ns3.p34.dynect.net type TXT returned 0 results
17:37:13.548989 DNS: Resolver: aspmx.l.google.com type TXT returned 0 results
17:37:13.549000 DNS: Resolver: aspmx2.googlemail.com type TXT returned 0 results
17:37:13.549014 DNS: Resolver: ns2.p34.dynect.net type CNAME returned 0 results
17:37:13.549032 DNS: Resolver: c.r06.twtrdns.net type TXT returned 0 results
17:37:13.549043 DNS: Resolver: d01-02.ns.twtrdns.net type CNAME returned 0 results
17:37:13.549063 DNS: Resolver: ns4.p34.dynect.net type TXT returned 0 results
17:37:13.549077 DNS: Resolver: d01-01.ns.twtrdns.net type TXT returned 0 results
17:37:13.549092 DNS: Resolver: alt2.aspmx.l.google.com type TXT returned 0 results
17:37:13.549116 DNS: Resolver: ns1.p34.dynect.net type TXT returned 0 results
17:37:13.549128 DNS: Resolver: b.r06.twtrdns.net type TXT returned 0 results
17:37:13.549139 DNS: Resolver: a.r06.twtrdns.net type AAAA returned 0 results
17:37:13.549151 DNS: Resolver: d01-02.ns.twtrdns.net type TXT returned 0 results
17:37:13.549169 DNS: Resolver: aspmx3.googlemail.com type TXT returned 0 results
17:37:13.549182 DNS: Resolver: d.r06.twtrdns.net type AAAA returned 0 results
17:37:13.549200 DNS: Resolver: ns2.p34.dynect.net type TXT returned 0 results
17:37:13.549211 DNS: Resolver: ns4.p34.dynect.net type AAAA returned 0 results
17:37:13.549236 DNS: Resolver: d01-01.ns.twtrdns.net type AAAA returned 0 results
17:37:13.549259 DNS: Resolver: c.r06.twtrdns.net type AAAA returned 0 results
17:37:13.549276 DNS: Resolver: b.r06.twtrdns.net type AAAA returned 0 results
17:37:13.549289 DNS: Resolver: d01-02.ns.twtrdns.net type AAAA returned 0 results
17:37:13.549307 DNS: Resolver: ns2.p34.dynect.net type AAAA returned 0 results
17:37:13.549319 DNS: Resolver: twitter.com type AAAA returned 0 results
17:37:15.548563 DNS: Resolver: _thirdparty.twitter.com type CNAME returned 0 results
17:37:15.548628 DNS: Resolver: _thirdparty.twitter.com type A returned 0 results
17:37:15.548649 DNS: Resolver: _thirdparty.twitter.com type AAAA returned 0 results
17:37:17.548517 DNS: Resolver: nwww.twitter.com type CNAME returned 0 results
17:37:17.548563 DNS: Resolver: nwww.twitter.com type TXT returned 0 results
17:37:17.548572 DNS: Resolver: nwww.twitter.com type A returned 0 results
17:37:17.548579 DNS: Resolver: nwww.twitter.com type AAAA returned 0 results
17:37:19.548733 DNS: Resolver: happening.mobile.twitter.com type CNAME returned 0 results
17:37:19.548790 DNS: Resolver: happening.mobile.twitter.com type TXT returned 0 results
17:37:19.548841 DNS: Resolver: happening.mobile.twitter.com type A returned 0 results
17:37:19.548890 DNS: Resolver: happening.mobile.twitter.com type AAAA returned 0 results
17:37:27.548577 DNS: Resolver: cards.twitter.com type TXT returned 0 results
17:37:27.548612 DNS: Resolver: cards.twitter.com type AAAA returned 0 results
17:37:29.548775 DNS: Resolver: studio.twitter.com type TXT returned 0 results
17:37:31.548960 DNS: Resolver: studio.twitter.com type AAAA returned 0 results

There are no type A lines for domains which do resolve (for example pic.twitter.com and mobile.twitter.com), only for domains which don't resolve. I'm not sure if that's expected or not.

I get the same results when specifying a resolver:

$ ./amass enum -v -ip -src -noalts -r 8.8.8.8 -include Pastebin -d twitter.com
Querying Pastebin for twitter.com subdomains
No names were discovered

Even though I can definitely resolve one of the subdomains via that resolver:

$ dig @8.8.8.8 mobile.twitter.com

; <<>> DiG 9.11.5-P4-5.1ubuntu2.1-Ubuntu <<>> @8.8.8.8 mobile.twitter.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47571
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;mobile.twitter.com.        IN  A

;; ANSWER SECTION:
mobile.twitter.com. 904 IN  A   104.244.42.198
mobile.twitter.com. 904 IN  A   104.244.42.6
mobile.twitter.com. 904 IN  A   104.244.42.70
mobile.twitter.com. 904 IN  A   104.244.42.134

;; Query time: 4 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Jan 18 17:50:36 GMT 2020
;; MSG SIZE  rcvd: 111
djcater commented 4 years ago

Strangely, the -r 8.8.8.8 doesn't seem to be doing what I expected, as there are still lots of queries going to other DNS servers. Here's some output from a Wireshark capture saved as CSV and sorted by number of queries to each server:

$ grep -i twitter twitter-8888.csv | grep 'query 0' | cut -d',' -f4 | tr -d '"' | sort -V | uniq -c | sort -rsn 
     99 64.6.65.6
     94 77.88.8.1
     85 1.1.1.1
     84 74.82.42.42
     82 1.0.0.1
     82 64.6.64.6
     81 8.8.8.8
     70 8.8.4.4

Looking specifically at mobile.twitter.com I see 3 A queries to 3 different servers (none of them 8.8.8.8), but all getting valid responses:

"3652","14.869998442","192.168.1.99","74.82.42.42","DNS","97","Standard query 0xe200 A mobile.twitter.com OPT"
"3659","14.872760823","74.82.42.42","192.168.1.99","DNS","153","Standard query response 0xe200 A mobile.twitter.com A 104.244.42.198 A 104.244.42.70 A 104.244.42.6 A 104.244.42.134 OPT"
"3665","14.875746222","192.168.1.99","64.6.65.6","DNS","97","Standard query 0xfa11 A mobile.twitter.com OPT"
"3677","14.890295294","64.6.65.6","192.168.1.99","DNS","161","Standard query response 0xfa11 A mobile.twitter.com A 104.244.42.6 A 104.244.42.198 A 104.244.42.70 A 104.244.42.134 OPT"
"3714","14.913741102","192.168.1.99","1.0.0.1","DNS","97","Standard query 0x7601 A mobile.twitter.com OPT"
"3719","14.918511004","1.0.0.1","192.168.1.99","DNS","153","Standard query response 0x7601 A mobile.twitter.com A 104.244.42.198 A 104.244.42.6 A 104.244.42.70 A 104.244.42.134 OPT"

Should I file that as a separate issue (-r not being respected) or is it related to this issue?

Flightkick commented 4 years ago

Not sure if https://github.com/OWASP/Amass/commit/8c7eb0ef2f809f91545895c3efc389b11ad71b36 should have fixed the -r flag. I just tried Amass 3.4.3, the -r flag still seems to be ignored.

─[✗]─[anon@parrot]─[~/Downloads/amass_v3.4.3_linux_amd64]
└──╼ $./amass intel -d example.com -whois -r 10.0.0.1 -v
SanityChecks: Resolver 1.0.0.1 failed to resolve good names
SanityChecks: Resolver 74.82.42.42 failed to resolve good names
SanityChecks: Resolver 64.6.65.6 failed to resolve good names
SanityChecks: Resolver 8.8.4.4 failed to resolve good names
SanityChecks: Resolver 64.6.64.6 failed to resolve good names
SanityChecks: Resolver 77.88.8.1 failed to resolve good names
SanityChecks: Resolver 8.8.8.8 failed to resolve good names
SanityChecks: Resolver 1.1.1.1 failed to resolve good names
^C
caffix commented 4 years ago

It was fixed for the 'enum' subcommand. Thank you for pointing this out

djcater commented 4 years ago

@caffix: Thanks for working on this, however unfortunately I still get the same results with version 3.4.4 (using the release binary from GitHub).

Exact same steps to reproduce as above. With -passive I get 19 results, but replacing -passive with -ip I get "No names were discovered". This is true even if I add --include-unresolvable - even the names which are NXDOMAIN do not show up.

$ ./amass enum -v -r 8.8.8.8 -passive -src -noalts -include Pastebin -d twitter.com 
Querying Pastebin for twitter.com subdomains
[Pastebin]        pic.twitter.com
[Pastebin]        mobile.twitter.com
[Pastebin]        happening.mobile.twitter.com
[Pastebin]        careers.twitter.com
[Pastebin]        www.twitter.com
[Pastebin]        search.twitter.com
[Pastebin]        support.twitter.com
[Pastebin]        about.twitter.com
[Pastebin]        blog.twitter.com
[Pastebin]        university.twitter.com
[Pastebin]        hoping.twitter.com
[Pastebin]        developer.twitter.com
[Pastebin]        1pic.twitter.com
[Pastebin]        tpic.twitter.com
[Pastebin]        connected.twitter.com
[Pastebin]        nwww.twitter.com
[Pastebin]        studio.twitter.com
[Pastebin]        ads-api.twitter.com
[Pastebin]        api.twitter.com

OWASP Amass v3.4.4                                https://github.com/OWASP/Amass
--------------------------------------------------------------------------------
19 names discovered - api: 19
$ rm -rv ~/.config/amass
$ ./amass enum -v -r 8.8.8.8 -ip -include-unresolvable -src -noalts -include Pastebin -d twitter.com 
Querying Pastebin for twitter.com subdomains
No names were discovered
$ cat ~/.config/amass/amass.log 
23:05:19.589436 Querying Pastebin for twitter.com subdomains
23:05:25.589517 DNS: DNS query on resolver 8.8.8.8, for careers.twitter.com type 16 returned 0 records
23:05:25.589567 DNS: DNS query on resolver 8.8.8.8, for 1pic.twitter.com type 5 returned error NXDOMAIN
23:05:25.589576 DNS: DNS query on resolver 8.8.8.8, for pic.twitter.com type 16 returned 0 records
23:05:25.589584 DNS: DNS query on resolver 8.8.8.8, for 1pic.twitter.com type 16 returned error NXDOMAIN
23:05:25.589607 DNS: DNS query on resolver 8.8.8.8, for careers.twitter.com type 28 returned 0 records
23:05:25.589611 DNS: DNS query on resolver 8.8.8.8, for pic.twitter.com type 28 returned 0 records
23:05:25.589615 DNS: DNS query on resolver 8.8.8.8, for 1pic.twitter.com type 1 returned error NXDOMAIN
23:05:25.589619 DNS: DNS query on resolver 8.8.8.8, for 1pic.twitter.com type 28 returned error NXDOMAIN
23:05:25.589624 DNS: DNS query on resolver 8.8.8.8, for developer.twitter.com type 16 returned 0 records
23:05:25.589628 DNS: DNS query on resolver 8.8.8.8, for www.twitter.com type 28 returned 0 records
23:05:25.589632 DNS: DNS query on resolver 8.8.8.8, for developer.twitter.com type 28 returned 0 records
23:05:25.589640 DNS: DNS query on resolver 8.8.8.8, for hoping.twitter.com type 5 returned error NXDOMAIN
23:05:25.589647 DNS: DNS query on resolver 8.8.8.8, for support.twitter.com type 5 returned 0 records
23:05:25.589651 DNS: DNS query on resolver 8.8.8.8, for university.twitter.com type 16 returned 0 records
23:05:25.589657 DNS: DNS query on resolver 8.8.8.8, for hoping.twitter.com type 16 returned error NXDOMAIN
23:05:25.589663 DNS: DNS query on resolver 8.8.8.8, for blog.twitter.com type 16 returned 0 records
23:05:25.589668 DNS: DNS query on resolver 8.8.8.8, for about.twitter.com type 16 returned 0 records
23:05:25.589673 DNS: DNS query on resolver 8.8.8.8, for search.twitter.com type 16 returned 0 records
23:05:25.589677 DNS: DNS query on resolver 8.8.8.8, for support.twitter.com type 16 returned 0 records
23:05:25.589684 DNS: DNS query on resolver 8.8.8.8, for hoping.twitter.com type 28 returned error NXDOMAIN
23:05:25.589689 DNS: SPF record query error: twitter.com: DNS query on resolver 8.8.8.8, for twitter.com type 99 returned 0 records
23:05:25.589694 DNS: DNS query on resolver 8.8.8.8, for hoping.twitter.com type 1 returned error NXDOMAIN
23:05:25.589700 DNS: DNS query on resolver 8.8.8.8, for university.twitter.com type 28 returned 0 records
23:05:25.589704 DNS: DNS query on resolver 8.8.8.8, for blog.twitter.com type 28 returned 0 records
23:05:25.589708 DNS: DNS query on resolver 8.8.8.8, for support.twitter.com type 28 returned 0 records
23:05:25.589712 DNS: DNS query on resolver 8.8.8.8, for about.twitter.com type 28 returned 0 records
23:05:25.589716 DNS: DNS query on resolver 8.8.8.8, for search.twitter.com type 28 returned 0 records
23:05:25.589721 DNS: DNS query on resolver 8.8.8.8, for mobile.twitter.com type 5 returned 0 records
23:05:25.589732 DNS: DNS query on resolver 8.8.8.8, for mobile.twitter.com type 16 returned 0 records
23:05:25.589736 DNS: DNS query on resolver 8.8.8.8, for mobile.twitter.com type 28 returned 0 records
23:05:27.589540 DNS: DNS query on resolver 8.8.8.8, for tpic.twitter.com type 5 returned error NXDOMAIN
23:05:27.589606 DNS: DNS query on resolver 8.8.8.8, for connected.twitter.com type 5 returned error NXDOMAIN
23:05:27.589616 DNS: DNS query on resolver 8.8.8.8, for connected.twitter.com type 16 returned error NXDOMAIN
23:05:27.589630 DNS: DNS query on resolver 8.8.8.8, for tpic.twitter.com type 16 returned error NXDOMAIN
23:05:27.589639 DNS: DNS query on resolver 8.8.8.8, for connected.twitter.com type 1 returned error NXDOMAIN
23:05:27.589647 DNS: DNS query on resolver 8.8.8.8, for tpic.twitter.com type 1 returned error NXDOMAIN
23:05:27.589652 DNS: DNS query on resolver 8.8.8.8, for tpic.twitter.com type 28 returned error NXDOMAIN
23:05:27.589660 DNS: DNS query on resolver 8.8.8.8, for connected.twitter.com type 28 returned error NXDOMAIN
23:05:27.589665 DNS: DNS query on resolver 8.8.8.8, for s.twitter.com type 5 returned 0 records
23:05:27.589670 DNS: DNS query on resolver 8.8.8.8, for twitter.com type 5 returned 0 records
23:05:27.589685 DNS: DNS query on resolver 8.8.8.8, for s.twitter.com type 16 returned 0 records
23:05:27.589690 DNS: DNS query on resolver 8.8.8.8, for s.twitter.com type 28 returned 0 records
23:05:27.589695 DNS: DNS query on resolver 8.8.8.8, for twitter.com type 28 returned 0 records
23:05:27.589700 DNS: DNS query on resolver 8.8.8.8, for aspmx2.googlemail.com type 5 returned 0 records
23:05:27.589705 DNS: DNS query on resolver 8.8.8.8, for alt1.aspmx.l.google.com type 5 returned 0 records
23:05:27.589710 DNS: DNS query on resolver 8.8.8.8, for alt2.aspmx.l.google.com type 5 returned 0 records
23:05:27.589715 DNS: DNS query on resolver 8.8.8.8, for aspmx3.googlemail.com type 5 returned 0 records
23:05:27.589722 DNS: DNS query on resolver 8.8.8.8, for b.r06.twtrdns.net type 5 returned 0 records
23:05:27.589729 DNS: DNS query on resolver 8.8.8.8, for aspmx3.googlemail.com type 16 returned 0 records
23:05:27.589736 DNS: DNS query on resolver 8.8.8.8, for ns2.p34.dynect.net type 5 returned 0 records
23:05:27.589742 DNS: DNS query on resolver 8.8.8.8, for alt1.aspmx.l.google.com type 16 returned 0 records
23:05:27.589750 DNS: DNS query on resolver 8.8.8.8, for aspmx.l.google.com type 5 returned 0 records
23:05:27.589755 DNS: DNS query on resolver 8.8.8.8, for aspmx2.googlemail.com type 16 returned 0 records
23:05:27.589764 DNS: DNS query on resolver 8.8.8.8, for alt2.aspmx.l.google.com type 16 returned 0 records
23:05:27.589769 DNS: DNS query on resolver 8.8.8.8, for b.r06.twtrdns.net type 16 returned 0 records
23:05:27.589774 DNS: DNS query on resolver 8.8.8.8, for d.r06.twtrdns.net type 5 returned 0 records
23:05:27.589778 DNS: DNS query on resolver 8.8.8.8, for ns3.p34.dynect.net type 5 returned 0 records
23:05:27.589783 DNS: DNS query on resolver 8.8.8.8, for aspmx.l.google.com type 16 returned 0 records
23:05:27.589788 DNS: DNS query on resolver 8.8.8.8, for c.r06.twtrdns.net type 5 returned 0 records
23:05:27.589793 DNS: DNS query on resolver 8.8.8.8, for d01-02.ns.twtrdns.net type 5 returned 0 records
23:05:27.589798 DNS: DNS query on resolver 8.8.8.8, for a.r06.twtrdns.net type 5 returned 0 records
23:05:27.589802 DNS: DNS query on resolver 8.8.8.8, for ns4.p34.dynect.net type 5 returned 0 records
23:05:27.589809 DNS: DNS query on resolver 8.8.8.8, for ns2.p34.dynect.net type 16 returned 0 records
23:05:27.589816 DNS: DNS query on resolver 8.8.8.8, for ns1.p34.dynect.net type 5 returned 0 records
23:05:27.589826 DNS: DNS query on resolver 8.8.8.8, for ns3.p34.dynect.net type 16 returned 0 records
23:05:27.589830 DNS: DNS query on resolver 8.8.8.8, for d.r06.twtrdns.net type 16 returned 0 records
23:05:27.589835 DNS: DNS query on resolver 8.8.8.8, for d01-02.ns.twtrdns.net type 16 returned 0 records
23:05:27.589840 DNS: DNS query on resolver 8.8.8.8, for a.r06.twtrdns.net type 16 returned 0 records
23:05:27.589844 DNS: DNS query on resolver 8.8.8.8, for d01-01.ns.twtrdns.net type 5 returned 0 records
23:05:27.589849 DNS: DNS query on resolver 8.8.8.8, for c.r06.twtrdns.net type 16 returned 0 records
23:05:27.589854 DNS: DNS query on resolver 8.8.8.8, for ns1.p34.dynect.net type 16 returned 0 records
23:05:27.589858 DNS: DNS query on resolver 8.8.8.8, for ns4.p34.dynect.net type 16 returned 0 records
23:05:27.589863 DNS: DNS query on resolver 8.8.8.8, for ns2.p34.dynect.net type 28 returned 0 records
23:05:27.589868 DNS: DNS query on resolver 8.8.8.8, for b.r06.twtrdns.net type 28 returned 0 records
23:05:27.589875 DNS: DNS query on resolver 8.8.8.8, for d01-01.ns.twtrdns.net type 16 returned 0 records
23:05:27.589880 DNS: DNS query on resolver 8.8.8.8, for d.r06.twtrdns.net type 28 returned 0 records
23:05:27.589885 DNS: DNS query on resolver 8.8.8.8, for c.r06.twtrdns.net type 28 returned 0 records
23:05:27.589890 DNS: DNS query on resolver 8.8.8.8, for d01-02.ns.twtrdns.net type 28 returned 0 records
23:05:27.589895 DNS: DNS query on resolver 8.8.8.8, for a.r06.twtrdns.net type 28 returned 0 records
23:05:27.589900 DNS: DNS query on resolver 8.8.8.8, for d01-01.ns.twtrdns.net type 28 returned 0 records
23:05:27.589907 DNS: DNS query on resolver 8.8.8.8, for ns4.p34.dynect.net type 28 returned 0 records
23:05:27.589913 DNS: DNS query on resolver 8.8.8.8, for _thirdparty.twitter.com type 5 returned 0 records
23:05:27.589918 DNS: DNS query on resolver 8.8.8.8, for _thirdparty.twitter.com type 1 returned 0 records
23:05:27.589923 DNS: DNS query on resolver 8.8.8.8, for _thirdparty.twitter.com type 28 returned 0 records
23:05:31.589560 DNS: DNS query on resolver 8.8.8.8, for happening.mobile.twitter.com type 16 returned error NXDOMAIN
23:05:31.589590 DNS: DNS query on resolver 8.8.8.8, for happening.mobile.twitter.com type 1 returned error NXDOMAIN
23:05:31.589603 DNS: DNS query on resolver 8.8.8.8, for happening.mobile.twitter.com type 5 returned error NXDOMAIN
23:05:31.589609 DNS: DNS query on resolver 8.8.8.8, for happening.mobile.twitter.com type 28 returned error NXDOMAIN
23:05:35.589619 DNS: DNS query on resolver 8.8.8.8, for nwww.twitter.com type 5 returned error NXDOMAIN
23:05:35.589682 DNS: DNS query on resolver 8.8.8.8, for nwww.twitter.com type 16 returned error NXDOMAIN
23:05:35.589697 DNS: DNS query on resolver 8.8.8.8, for nwww.twitter.com type 1 returned error NXDOMAIN
23:05:35.589724 DNS: DNS query on resolver 8.8.8.8, for nwww.twitter.com type 28 returned error NXDOMAIN
23:05:41.589755 DNS: DNS query on resolver 8.8.8.8, for studio.twitter.com type 28 returned 0 records
23:05:41.589868 DNS: DNS query on resolver 8.8.8.8, for studio.twitter.com type 16 returned 0 records
23:05:41.589900 DNS: DNS query on resolver 8.8.8.8, for api.twitter.com type 5 returned 0 records
23:05:41.589960 DNS: DNS query on resolver 8.8.8.8, for api.twitter.com type 16 returned 0 records
23:05:41.589982 DNS: DNS query on resolver 8.8.8.8, for ads-api.twitter.com type 16 returned 0 records
23:05:41.590001 DNS: DNS query on resolver 8.8.8.8, for ads-api.twitter.com type 28 returned 0 records
23:05:41.590039 DNS: DNS query on resolver 8.8.8.8, for api.twitter.com type 28 returned 0 records

The -r option does now work though, which is good (#363), however trying different resolvers makes no difference to this issue (tried 1.1.1.1, 8.8.8.8, and my ISP's DNS servers).

I get the same results with and without a VPN, and both on the Ubuntu 19.10 host and with the latest caffix/amass Docker image.

I confirmed that manually querying those DNS servers with dig for mobile.twitter.com does give valid A records back, so the DNS servers are reachable and functioning, and on top of that, during the Amass run I can see the correct DNS responses in Wireshark! But for some reason they are getting ignored.

Please let me know if there's anything I can do to continue debugging this.

caffix commented 4 years ago

The issues have been fixed. Until we implement the entry of partial information into the database, you’ll need to provide the enumeration a data source for IP/Netblock/ASN lookups.

In addition to Pastebin, add NetworksDB and/or ShadowServer to handle this requirement

djcater commented 4 years ago

Ah, OK, thanks. Showing "No names were discovered" is very confusing then - I did not realise that one of NetworksDB or ShadowServer was mandatory just to be able to display the names which resolve. It would be better to error if a mandatory source is missing than to pretend that no names resolved.

caffix commented 4 years ago

Agreed, and we recently discussed this issue. The entry of partial information into the graph database will solve this problem. Thank you again for your assistance!