search

zmap / zdns

Fast DNS Lookup Library and CLI Tool
Apache License 2.0
939 stars 123 forks source link

Move ZDNS to using ZFlags instead of Cobra #427

Closed phillip-stephens closed 2 months ago

phillip-stephens commented 2 months ago

Description

ZFlags should give us more flexibility (we can now use short form flags (-t!)) and make ZDNS act more similar to ZGrab. Additionally, it should make having multiple modules as requested #234 possible, similar to how it's done in ZGrab.

Version Flag (-v, --version)

$ ./zdns -v
zdns version 1.1.0

New --help

$ ./zdns -h
Usage:
  zdns [OPTIONS] <command>

ZDNS is a library and CLI tool for making very fast DNS requests. It's built upon
https://github.com/zmap/dns (and in turn https://github.com/miekg/dns) for constructing
and parsing raw DNS packets.
ZDNS also includes its own recursive resolution and a cache to further optimize performance.

General Options:
      --all-nameservers        Perform the lookup via all the nameservers for the domain.
      --cache-size=            how many items can be stored in internal recursive cache (default: 10000)
      --go-processes=          number of OS processes (GOMAXPROCS by default) (default: 0)
      --iteration-timeout=     timeout for a single iterative step in an iterative query, in seconds. Only applicable with --iterative (default: 4)
      --iterative              Perform own iteration instead of relying on recursive resolver
      --max-depth=             how deep should we recurse when performing iterative lookups (default: 10)
      --name-server-mode       Treats input as nameservers to query with a static query rather than queries to send to a static name server
      --name-servers=          List of DNS servers to use. Can be passed as comma-delimited string or via @/path/to/file. If no port is specified, defaults to 53.
      --nanoseconds            Use nanosecond resolution timestamps in output
      --no-follow-cnames       do not follow CNAMEs/DNAMEs in the lookup process
      --retries=               how many times should zdns retry query if timeout or temporary failure (default: 1)
  -t, --threads=               number of lightweight go threads (default: 1000)
      --timeout=               timeout for resolving a individual name, in seconds (default: 15)
  -v, --version                Print the version of zdns and exit

Query Options:
      --checking-disabled      Sends DNS packets with the CD bit set
      --class=                 DNS class to query. Options: INET, CSNET, CHAOS, HESIOD, NONE, ANY. (default: INET)
      --client-subnet=         Client subnet in CIDR format for EDNS0.
      --dnssec                 Requests DNSSEC records by setting the DNSSEC OK (DO) bit
      --nsid                   Request NSID.

Network Options:
      --4                      utilize IPv4 query transport only, incompatible with --6
      --6                      utilize IPv6 query transport only, incompatible with --4
      --local-addr=            comma-delimited list of local addresses to use, serve as the source IP for outbound queries
      --local-interface=       local interface to use
      --no-recycle-sockets     do not create long-lived unbound UDP socket for each thread at launch and reuse for all (UDP) queries
      --prefer-ipv4-iteration  Prefer IPv4/A record lookups during iterative resolution. Ignored unless used with both IPv4 and IPv6 query transport
      --prefer-ipv6-iteration  Prefer IPv6/AAAA record lookups during iterative resolution. Ignored unless used with both IPv4 and IPv6 query transport
      --tcp-only               Only perform lookups over TCP
      --udp-only               Only perform lookups over UDP

Input/Output Options:
      --alexa                  is input file from Alexa Top Million download
      --blacklist-file=        blacklist file for servers to exclude from lookups
      --conf-file=             config file for DNS servers (default: /etc/resolv.conf)
      --include-fields=        Comma separated list of fields to additionally output beyond result verbosity. Options: class, protocol, ttl, resolver, flags
  -f, --input-file=            names to read, defaults to stdin (default: -)
      --log-file=              where should JSON logs be saved, defaults to stderr (default: -)
      --metadata-file=         where should JSON metadata be saved, defaults to no metadata output. Use '-' for stderr.
      --metadata-passthrough   if input records have the form 'name,METADATA', METADATA will be propagated to the output
  -o, --output-file=           where should JSON output be saved, defaults to stdout (default: -)
      --override-name=         name overrides all passed in names. Commonly used with --name-server-mode.
      --prefix=                name to be prepended to what's passed in (e.g., www.)
      --result-verbosity=      Sets verbosity of each output record. Options: short, normal, long, trace (default: normal)
      --verbosity=             log verbosity: 1 (lowest)--5 (highest) (default: 3)

Help Options:
  -h, --help                   Show this help message

Available commands:
  A
  AAAA
  AFSDB
  ALOOKUP
  ANY
  ATMA
  AVC
  AXFR
  BINDVERSION
  CAA
  CDNSKEY
  CDS
  CERT
  CNAME
  CSYNC
  DHCID
  DMARC
  DNAME
  DNSKEY
  DS
  EID
  EUI48
  EUI64
  GID
  GPOS
  HINFO
  HIP
  HTTPS
  ISDN
  KEY
  KX
  L32
  L64
  LOC
  LP
  MB
  MD
  MF
  MG
  MR
  MX
  MXLOOKUP
  NAPTR
  NID
  NIMLOC
  NINFO
  NS
  NSAPPTR
  NSEC
  NSEC3
  NSEC3PARAM
  NSLOOKUP
  NULL
  NXT
  OPENPGPKEY
  PTR
  PX
  RP
  RRSIG
  RT
  SMIMEA
  SOA
  SPF
  SPF
  SRV
  SSHFP
  SVCB
  TALINK
  TKEY
  TLSA
  TXT
  UID
  UINFO
  UNSPEC
  URI

Module-Specific Help

$ ./zdns alookup --help
Usage:
  zdns [OPTIONS] ALOOKUP [ALOOKUP-OPTIONS]

alookup will get the information that is typically desired, instead of just the information that exists in a single record. Specifically, alookup acts similar to nslookup and will follow CNAME records.

General Options:
      --all-nameservers        Perform the lookup via all the nameservers for the domain.
      --cache-size=            how many items can be stored in internal recursive cache (default: 10000)
      --go-processes=          number of OS processes (GOMAXPROCS by default) (default: 0)
      --iteration-timeout=     timeout for a single iterative step in an iterative query, in seconds. Only applicable with --iterative (default: 4)
      --iterative              Perform own iteration instead of relying on recursive resolver
      --max-depth=             how deep should we recurse when performing iterative lookups (default: 10)
      --name-server-mode       Treats input as nameservers to query with a static query rather than queries to send to a static name server
      --name-servers=          List of DNS servers to use. Can be passed as comma-delimited string or via @/path/to/file. If no port is specified, defaults to 53.
      --nanoseconds            Use nanosecond resolution timestamps in output
      --no-follow-cnames       do not follow CNAMEs/DNAMEs in the lookup process
      --retries=               how many times should zdns retry query if timeout or temporary failure (default: 1)
  -t, --threads=               number of lightweight go threads (default: 1000)
      --timeout=               timeout for resolving a individual name, in seconds (default: 15)
  -v, --version                Print the version of zdns and exit

Query Options:
      --checking-disabled      Sends DNS packets with the CD bit set
      --class=                 DNS class to query. Options: INET, CSNET, CHAOS, HESIOD, NONE, ANY. (default: INET)
      --client-subnet=         Client subnet in CIDR format for EDNS0.
      --dnssec                 Requests DNSSEC records by setting the DNSSEC OK (DO) bit
      --nsid                   Request NSID.

Network Options:
      --4                      utilize IPv4 query transport only, incompatible with --6
      --6                      utilize IPv6 query transport only, incompatible with --4
      --local-addr=            comma-delimited list of local addresses to use, serve as the source IP for outbound queries
      --local-interface=       local interface to use
      --no-recycle-sockets     do not create long-lived unbound UDP socket for each thread at launch and reuse for all (UDP) queries
      --prefer-ipv4-iteration  Prefer IPv4/A record lookups during iterative resolution. Ignored unless used with both IPv4 and IPv6 query transport
      --prefer-ipv6-iteration  Prefer IPv6/AAAA record lookups during iterative resolution. Ignored unless used with both IPv4 and IPv6 query transport
      --tcp-only               Only perform lookups over TCP
      --udp-only               Only perform lookups over UDP

Input/Output Options:
      --alexa                  is input file from Alexa Top Million download
      --blacklist-file=        blacklist file for servers to exclude from lookups
      --conf-file=             config file for DNS servers (default: /etc/resolv.conf)
      --include-fields=        Comma separated list of fields to additionally output beyond result verbosity. Options: class, protocol, ttl, resolver, flags
  -f, --input-file=            names to read, defaults to stdin (default: -)
      --log-file=              where should JSON logs be saved, defaults to stderr (default: -)
      --metadata-file=         where should JSON metadata be saved, defaults to no metadata output. Use '-' for stderr.
      --metadata-passthrough   if input records have the form 'name,METADATA', METADATA will be propagated to the output
  -o, --output-file=           where should JSON output be saved, defaults to stdout (default: -)
      --override-name=         name overrides all passed in names. Commonly used with --name-server-mode.
      --prefix=                name to be prepended to what's passed in (e.g., www.)
      --result-verbosity=      Sets verbosity of each output record. Options: short, normal, long, trace (default: normal)
      --verbosity=             log verbosity: 1 (lowest)--5 (highest) (default: 3)

Help Options:
  -h, --help                   Show this help message

[ALOOKUP command options]
          --ipv4-lookup        perform A lookups for each server
          --ipv6-lookup        perform AAAA lookups for each server

Testing

Normal Usage

$ echo "cloudflare.com" | ./zdns A
{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":512,"version":0}],"answers":[{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":164,"type":"A"},{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":164,"type":"A"}],"protocol":"udp","resolver":"[2603:6013:9d00:3302::1]:53"},"duration":0.04202225,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-20T16:13:44-04:00"}

Using a flag

$ echo "cloudflare.com" | ./zdns A --verbosity=5
INFO[0000] No iteration IP preference specified, defaulting to IPv4 preferred. See --prefer-ipv4-iteration and --prefer-ipv6-iteration for more info
INFO[0000] using local addresses: [192.168.1.243 2603:6013:9d00:3302:c460:464e:f5c0:9930]
INFO[0000] for non-iterative lookups, using external nameservers: 192.168.1.1:53, [2603:6013:9d00:3302::1]:53
INFO[0000] for iterative lookups, using nameservers: 192.168.1.1:53, [2603:6013:9d00:3302::1]:53
INFO[0000] no name server provided for external lookup, using  random external name server: [2603:6013:9d00:3302::1]:53
DEBU[0000] DEPTH 00:[MIEKG-IN: starting a C/DNAME following lookup for  cloudflare.com  ( 1 )]
DEBU[0000] DEPTH 01:    [MIEKG-IN: following external lookup for  cloudflare.com  ( 1 )]
DEBU[0000] DEPTH 01:    [****WIRE LOOKUP***  A   cloudflare.com   [2603:6013:9d00:3302::1]:53]
DEBU[0000] DEPTH 01:    [MIEKG-OUT: following external lookup for  cloudflare.com  ( 1 ) with  1  attempts: status:  NOERROR  , err:  <nil>]
{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":4096,"version":0}],"answers":[{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":153,"type":"A"},{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":153,"type":"A"}],"protocol":"udp","resolver":"[2603:6013:9d00:3302::1]:53"},"duration":0.019543083,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-20T16:13:55-04:00"}

Using a flag before a module

$ make && echo "cloudflare.com" | ./zdns --verbosity 5 alookup       
go build -o zdns
INFO[0000] No iteration IP preference specified, defaulting to IPv4 preferred. See --prefer-ipv4-iteration and --prefer-ipv6-iteration for more info 
INFO[0000] using local addresses: [192.168.1.243 2603:6013:9d00:3302:c460:464e:f5c0:9930] 
INFO[0000] for non-iterative lookups, using external nameservers: 192.168.1.1:53, [2603:6013:9d00:3302::1]:53 
INFO[0000] for iterative lookups, using nameservers: 192.168.1.1:53, [2603:6013:9d00:3302::1]:53 
INFO[0000] no name server provided for external lookup, using  random external name server: [2603:6013:9d00:3302::1]:53 
DEBU[0000] DEPTH 00:[MIEKG-IN: starting a C/DNAME following lookup for  cloudflare.com  ( 1 )] 
DEBU[0000] DEPTH 01:    [MIEKG-IN: following external lookup for  cloudflare.com  ( 1 )] 
DEBU[0000] DEPTH 01:    [****WIRE LOOKUP***  A   cloudflare.com   [2603:6013:9d00:3302::1]:53] 
DEBU[0000] DEPTH 01:    [MIEKG-OUT: following external lookup for  cloudflare.com  ( 1 ) with  1  attempts: status:  NOERROR  , err:  <nil>] 
{"data":{"ipv4_addresses":["104.16.133.229","104.16.132.229"]},"duration":0.037664875,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-21T12:15:43-04:00"}

Using a special module that adds extra flags

$ echo "cloudflare.com" | ./zdns nslookup --ipv4-lookup
{"data":{"servers":[{"ipv4_addresses":["162.159.5.6","162.159.3.11"],"name":"ns6.cloudflare.com","ttl":86400,"type":"NS"},{"ipv4_addresses":["162.159.4.8","162.159.6.6"],"name":"ns7.cloudflare.com","ttl":86400,"type":"NS"},{"ipv4_addresses":["162.159.0.33","162.159.7.226"],"name":"ns3.cloudflare.com","ttl":86400,"type":"NS"},{"ipv4_addresses":["162.159.8.55","162.159.1.33"],"name":"ns4.cloudflare.com","ttl":86400,"type":"NS"},{"ipv4_addresses":["162.159.9.55","162.159.2.9"],"name":"ns5.cloudflare.com","ttl":86400,"type":"NS"}]},"duration":0.231573292,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-20T16:15:29-04:00"}

~/zdns on  phillip/zflags-cli! ⌚ 16:15:29
$ echo "cloudflare.com" | ./zdns nslookup --ipv6-lookup
{"data":{"servers":[{"ipv6_addresses":["2400:cb00:2049:1::a29f:209","2400:cb00:2049:1::a29f:937"],"name":"ns5.cloudflare.com","ttl":86395,"type":"NS"},{"ipv6_addresses":["2400:cb00:2049:1::a29f:506","2400:cb00:2049:1::a29f:30b"],"name":"ns6.cloudflare.com","ttl":86395,"type":"NS"},{"ipv6_addresses":["2400:cb00:2049:1::a29f:408","2400:cb00:2049:1::a29f:606"],"name":"ns7.cloudflare.com","ttl":86395,"type":"NS"},{"ipv6_addresses":["2400:cb00:2049:1::a29f:7e2","2400:cb00:2049:1::a29f:21"],"name":"ns3.cloudflare.com","ttl":86395,"type":"NS"},{"ipv6_addresses":["2400:cb00:2049:1::a29f:837","2400:cb00:2049:1::a29f:121"],"name":"ns4.cloudflare.com","ttl":86395,"type":"NS"}]},"duration":0.331910166,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-20T16:15:34-04:00"}
phillip-stephens commented 2 months ago

Noticed I need to add -v, --version support. I'll do that and then open it for review.