Consider lowering DNS resolve distance

TheTechromancer commented 8 months ago

By default BBOT resolves out to a distance of 2, but since the main purpose of resolving to that distance is to find things like A->PTR->A, and since I believe the distance-2 events also get resolved for the purpose of scoping (even though they aren't emitted), it would probably be okay to decrease the distance to 1. This would speed up scans without sacrificing thoroughness.

TheTechromancer commented 8 months ago

Mocking DNS with the following entries:

scanner.helpers.dns.mock_dns({
    ("one.com", "TXT"): "two.com",
    ("two.com", "TXT"): "three.com",
    ("three.com", "TXT"): "four.com",
    ("four.com", "TXT"): "five.com",
})

Results in these DNS queries (with default distance settings):

[WARN] one.com:{'type': 'A', 'use_cache': True}
[WARN] one.com:{'type': 'AAAA', 'use_cache': True}
[WARN] one.com:{'type': 'SRV', 'use_cache': True}
[WARN] one.com:{'type': 'MX', 'use_cache': True}
[WARN] one.com:{'type': 'NS', 'use_cache': True}
[WARN] one.com:{'type': 'SOA', 'use_cache': True}
[WARN] one.com:{'type': 'CNAME', 'use_cache': True}
[WARN] one.com:{'type': 'TXT', 'use_cache': True}
[DNS_NAME]              one.com TARGET  (domain, in-scope, resolved, target, txt-record)
[WARN] two.com:{'type': 'A', 'use_cache': True}
[WARN] two.com:{'type': 'AAAA', 'use_cache': True}
[WARN] two.com:{'type': 'SRV', 'use_cache': True}
[WARN] two.com:{'type': 'MX', 'use_cache': True}
[WARN] two.com:{'type': 'NS', 'use_cache': True}
[WARN] two.com:{'type': 'SOA', 'use_cache': True}
[WARN] two.com:{'type': 'CNAME', 'use_cache': True}
[WARN] two.com:{'type': 'TXT', 'use_cache': True}
[DNS_NAME]              two.com TXT (affiliate, distance-1, domain, resolved, txt-record)
[WARN] three.com:{'type': 'A', 'use_cache': True}
[WARN] three.com:{'type': 'AAAA', 'use_cache': True}
[WARN] three.com:{'type': 'SRV', 'use_cache': True}
[WARN] three.com:{'type': 'MX', 'use_cache': True}
[WARN] three.com:{'type': 'NS', 'use_cache': True}
[WARN] three.com:{'type': 'SOA', 'use_cache': True}
[WARN] three.com:{'type': 'CNAME', 'use_cache': True}
[WARN] three.com:{'type': 'TXT', 'use_cache': True}
[DNS_NAME]              three.com   TXT (distance-2, domain, resolved, txt-record)

With an in-scope host at distance-3:

scanner.helpers.dns.mock_dns({
    ("one.com", "TXT"): "two.com",
    ("two.com", "TXT"): "three.com",
    ("three.com", "TXT"): "four.one.com",
    ("four.one.com", "A"): "1.2.3.4",
})

It successfully finds it:

[WARN] one.com:{'type': 'A', 'use_cache': True}
[WARN] one.com:{'type': 'AAAA', 'use_cache': True}
[WARN] one.com:{'type': 'SRV', 'use_cache': True}
[WARN] one.com:{'type': 'MX', 'use_cache': True}
[WARN] one.com:{'type': 'NS', 'use_cache': True}
[WARN] one.com:{'type': 'SOA', 'use_cache': True}
[WARN] one.com:{'type': 'CNAME', 'use_cache': True}
[WARN] one.com:{'type': 'TXT', 'use_cache': True}
[DNS_NAME]              one.com TARGET  (domain, in-scope, resolved, target, txt-record)
[WARN] two.com:{'type': 'A', 'use_cache': True}
[WARN] two.com:{'type': 'AAAA', 'use_cache': True}
[WARN] two.com:{'type': 'SRV', 'use_cache': True}
[WARN] two.com:{'type': 'MX', 'use_cache': True}
[WARN] two.com:{'type': 'NS', 'use_cache': True}
[WARN] two.com:{'type': 'SOA', 'use_cache': True}
[WARN] two.com:{'type': 'CNAME', 'use_cache': True}
[WARN] two.com:{'type': 'TXT', 'use_cache': True}
[DNS_NAME]              two.com TXT (affiliate, distance-1, domain, resolved, txt-record)
[ORG_STUB]              one speculate   (distance-1)
[WARN] three.com:{'type': 'A', 'use_cache': True}
[WARN] three.com:{'type': 'AAAA', 'use_cache': True}
[WARN] three.com:{'type': 'SRV', 'use_cache': True}
[WARN] three.com:{'type': 'MX', 'use_cache': True}
[WARN] three.com:{'type': 'NS', 'use_cache': True}
[WARN] three.com:{'type': 'SOA', 'use_cache': True}
[WARN] three.com:{'type': 'CNAME', 'use_cache': True}
[WARN] three.com:{'type': 'TXT', 'use_cache': True}
[DNS_NAME]              three.com   TXT (distance-2, domain, resolved, txt-record)
[WARN] four.one.com:{'type': 'A', 'use_cache': True}
[WARN] four.one.com:{'type': 'AAAA', 'use_cache': True}
[WARN] four.one.com:{'type': 'SRV', 'use_cache': True}
[WARN] four.one.com:{'type': 'MX', 'use_cache': True}
[WARN] four.one.com:{'type': 'NS', 'use_cache': True}
[WARN] four.one.com:{'type': 'SOA', 'use_cache': True}
[WARN] four.one.com:{'type': 'CNAME', 'use_cache': True}
[WARN] four.one.com:{'type': 'TXT', 'use_cache': True}
[WARN] four.one.com:{'type': 'A', 'use_cache': True}
[WARN] four.one.com:{'type': 'AAAA', 'use_cache': True}
[WARN] four.one.com:{'type': 'SRV', 'use_cache': True}
[WARN] four.one.com:{'type': 'MX', 'use_cache': True}
[WARN] four.one.com:{'type': 'NS', 'use_cache': True}
[WARN] four.one.com:{'type': 'SOA', 'use_cache': True}
[WARN] four.one.com:{'type': 'CNAME', 'use_cache': True}
[WARN] four.one.com:{'type': 'TXT', 'use_cache': True}
[DNS_NAME]              four.one.com    TXT (a-record, in-scope, resolved, subdomain)
[WARN] 1.2.3.4:{'type': 'PTR', 'use_cache': True}
[IP_ADDRESS]            1.2.3.4 A   (distance-1, ipv4)

Confirmed that the purpose-built case host->IP->host is still detected even with scope_dns_search_distance=1:

scanner.helpers.dns.mock_dns({
    ("one.com", "A"): "1.2.3.4",
    ("1.2.3.4", "PTR"): "three.one.com",
    ("three.one.com", "A"): "4.3.2.1",
})

[WARN] one.com:{'type': 'A', 'use_cache': True}
[WARN] one.com:{'type': 'AAAA', 'use_cache': True}
[WARN] one.com:{'type': 'SRV', 'use_cache': True}
[WARN] one.com:{'type': 'MX', 'use_cache': True}
[WARN] one.com:{'type': 'NS', 'use_cache': True}
[WARN] one.com:{'type': 'SOA', 'use_cache': True}
[WARN] one.com:{'type': 'CNAME', 'use_cache': True}
[WARN] one.com:{'type': 'TXT', 'use_cache': True}
[DNS_NAME]              one.com TARGET  (a-record, domain, in-scope, resolved, target)
[WARN] 1.2.3.4:{'type': 'PTR', 'use_cache': True}
[IP_ADDRESS]            1.2.3.4 A   (distance-1, ipv4, ptr-record, resolved)
[WARN] three.one.com:{'type': 'A', 'use_cache': True}
[WARN] three.one.com:{'type': 'AAAA', 'use_cache': True}
[WARN] three.one.com:{'type': 'SRV', 'use_cache': True}
[WARN] three.one.com:{'type': 'MX', 'use_cache': True}
[WARN] three.one.com:{'type': 'NS', 'use_cache': True}
[WARN] three.one.com:{'type': 'SOA', 'use_cache': True}
[WARN] three.one.com:{'type': 'CNAME', 'use_cache': True}
[WARN] three.one.com:{'type': 'TXT', 'use_cache': True}
[WARN] three.one.com:{'type': 'A', 'use_cache': True}
[WARN] three.one.com:{'type': 'AAAA', 'use_cache': True}
[WARN] three.one.com:{'type': 'SRV', 'use_cache': True}
[WARN] three.one.com:{'type': 'MX', 'use_cache': True}
[WARN] three.one.com:{'type': 'NS', 'use_cache': True}
[WARN] three.one.com:{'type': 'SOA', 'use_cache': True}
[WARN] three.one.com:{'type': 'CNAME', 'use_cache': True}
[WARN] three.one.com:{'type': 'TXT', 'use_cache': True}
[DNS_NAME]              three.one.com   PTR (a-record, in-scope, resolved, subdomain)
[WARN] 4.3.2.1:{'type': 'PTR', 'use_cache': True}

TheTechromancer commented 8 months ago

I feel comfortable decreasing the distance, but only if it actually affects the speed of the scan.

Benchmarking bbot -t tesla.com -m subdomaincenter httpx on Linode.

With scope_dns_search_distance=2: 2m43.161s, 2742 events, in-scope subdomains: 441
With scope_dns_search_distance=1: 2m47.727s, 2189 events, in-scope subdomains: 441

httpx takes up the majority of the time here so it's impossible to see the time difference, but the difference in accuracy is pretty clear.

Let's try again on ebay.com -t azure_tenant subdomaincenter:

With scope_dns_search_distance=2: 2m55.739s, 5589 events, in-scope subdomains: 2654
With scope_dns_search_distance=1: 2m47.083s, 4977 events, in-scope subdomains: 2655

TheTechromancer commented 8 months ago

Implemented in https://github.com/blacklanternsecurity/bbot/pull/987.

blacklanternsecurity / bbot

Consider lowering DNS resolve distance #950