search

owasp-amass / amass

In-depth attack surface mapping and asset discovery
https://owasp.org/www-project-amass/
Other
11.63k stars 1.85k forks source link

Consistent (and inconsistent) false-negatives on known existing domains #47

Closed dtph closed 5 years ago

dtph commented 5 years ago

Hello!

There appear to be issues with the domain discovery process. amass was thrown at scanme.nmap.org to see what domains would be discovered from there. Naturally it didn't find much-- there aren't any subdomains there. After that was fruitless, it was thrown at nmap.org, and to my surprise, it found almost nothing. Not even www.nmap.org.

After some tweaking, I decided to try changing the nameservers, other than the default DigitalOcean nameservers I was using. This seemed to find more domains, but not nearly as many as other domain enumeration tools. I decided to enable logging, and to my shock, it was consistently assuming that certain domain names did not resolve. Of note, scanme.nmap.org, according to amass, did not resolve.

Of interest as well is that the nameservers chosen seems to affect the outcome of the result. Scanning entirely with -r 4.2.2.2,4.2.2.3 resolves a few domains correctly (though still yields false negatives), yet using -r 8.8.8.8,9.9.9.9,1.1.1.1 seems to resolve nothing. Going further, hitting the nameservers specific to nmap.org-- namely, ns[1-5].linode.com and hostmaster.insecure.org, additionally don't seem to yield anything.

There seem to be some more problems in here, such as improper parsing of URLs it looks like from just the error messages? But I'm not certain. Either way, here are some notable highlights of false negatives of known active domains.

Attached are logs of some scans. Here's the table of the commands run to produce them. Hope all this helps!

Data

System information

uname -a: Linux sputnik 4.9.0-7-amd64 #1 SMP Debian 4.9.110-1 (2018-07-05) x86_64 GNU/Linux amass -version: version 2.8.1 Installed with: snap install amass

Execution information

Scan information

amass-level3.log: amass -v -active -brute -ip -r 4.2.2.2,4.2.2.3 -log amass-level3.log -d nmap.org amass-google.log: amass -v -active -brute -ip -r 8.8.8.8,9.9.9.9 -log amass-google.log -d nmap.org amass-cloudflare.log: amass -v -active -brute -ip -r 1.1.1.1 -log amass-cloudflare.log -d nmap.org amass-direct-ns.log: amass -v -active -brute -ip -r 162.159.27.72,45.33.49.119 -d nmap.org

Notable false-negatives

amass-cloudflare.log amass-direct-ns.log amass-google.log amass-level3.log

caffix commented 5 years ago

@dtph Thank you very much for the excellent feedback! Some of the problems you experienced was due to the current wildcard detection, which I'm still working to improve further. When version 2.8.2 is released, try again, and I think it will have helped the false-negatives problem. Let me know what you think.

dtph commented 5 years ago

I'll give 2.8.2 a test-drive! In the meantime we found a binary that doesn't seem to exhibit these issues, so we're falling back on that. Thanks for the quick reply! <3

bkkgbkjb commented 5 years ago

I'm also bothered by this problem many times before.

Din't fire an issue because I could not believe this would happen at a software with 2.x.y version in 2018...

Anyway, please solve this issue quickly since amass is so excellent in cooperating with other network mapping softwares; but now it is not fully reliable

BTW, @dtph would you please show me the version of binary file you're using and how I can get it? I would give it an immediate try

Thanks :)

caffix commented 5 years ago

@bkkgbkjb Can you please be more specific with your experience with the current version of Amass? This issue has already been addressed, and I am simply waiting for @dtph to confirm that the problem has been fixed.

bkkgbkjb commented 5 years ago

OK

I will inspect into it this weekend

At that time, I will open a new issue if needed

bkkgbkjb commented 5 years ago

Sorry to say I still found some strange results from using amass to do simple enumerations based on given dict

By using amass 2.8.8 installed from go get and command amass -ip -src -ef exclude.txt -brute -w tools/dict.txt -d cggc.cn I got only one returned result(mail is not in my dict.txt):

[Forward DNS]     mail.cggc.cn,58.49.25.138,2408:864e:401::
Average DNS names processed: 1/sec
Average DNS names processed: 0/sec
Average DNS names processed: 0/sec

OWASP Amass v2.8.8                                https://github.com/OWASP/Amass
--------------------------------------------------------------------------------
1 names discovered - dns: 1
--------------------------------------------------------------------------------
ASN: 4837 - CHINA169-BACKBONE CHINA UNICOM China169 Backbone, CN
    2408:864e::/32      1    Subdomain Name(s)
ASN: 4134 - CHINANET-BACKBONE No.31,Jin-rong Street, CN
    58.48.0.0/13        1    Subdomain Name(s)

However, a more simple software gobuster with cmd gobuster -t 15 -i -w tools/dict.txt -u cggc.cn -m dns would feedback a different result:

=====================================================
Gobuster v2.0.1              OJ Reeves (@TheColonial)
=====================================================
[+] Mode         : dns
[+] Url/Domain   : cggc.cn
[+] Threads      : 15
[+] Wordlist     : tools/dict.txt
=====================================================
2019/01/10 08:54:35 Starting gobuster
=====================================================
2019/01/10 08:54:36 [-] Unable to validate base domain: cggc.cn
Found: oos.cggc.cn [58.49.25.169]
=====================================================
2019/01/10 08:54:36 Finished
=====================================================

the content of dict.txt can be any as long as including oos

bkkgbkjb commented 5 years ago

Thanks? @caffix

caffix commented 5 years ago

mail.cggc.cn was discovered via DNS, not using your wordlist. The problem you are experiencing with brute forcing has been fixed in the develop branch and will be in the next release. Thank you!

caffix commented 5 years ago

@dtph Before I close this issue, I thought it would be helpful to mention that a log entry saying that a DNS query for a name with record type 5 or 16 resulting in 0 records is not a false negative when you then dig for the name's A record. The DNS record of type A has the numerical type of 1. Types 5 and 16 are CNAME and TXT, respectively. Hope this helps