search

DNS-OARC / dsc

DNS Statistics Collector
https://www.dns-oarc.net/oarc/data/dsc
Other
109 stars 25 forks source link

Wrong TLDs when Public Suffix List is enabled: #285

Closed klaus-nicat closed 1 year ago

klaus-nicat commented 1 year ago

I think we found a bug in DSC. Some time ago you implemented the PublicSuffixList in DSC (config option tldlist), to treat 3rd level domains like example.co.uk as 2nd level domain. Just recently we activated that feature, and noticed that the Country-to-TLD analyzer also changed its behavior:

Without PSL:

<array name="countries" dimensions="2" start_time="1683724320" stop_time="1683724380">
  <dimension number="1" type="Countries"/>
  <dimension number="2" type="TLD"/>
  <data>
    <Countries val="BA">
      <TLD val="com" count="83"/>
      <TLD val="de" count="30"/>
      <TLD val="net" count="25"/>
      <TLD val="at" count="22"/>
      <TLD val="eu" count="8"/>
      <TLD val="no" count="6"/>
      <TLD val="org" count="6"/>
      <TLD val="africa" count="6"/>
      <TLD val="cz" count="3"/>
      <TLD val="hr" count="3"/>
      <TLD val="info" count="3"/>
      <TLD val="tv" count="2"/>
      <TLD val="it" count="2"/>
      <TLD val="co" count="1"/>
      <TLD val="ba" count="1"/>
      <TLD val="ch" count="1"/>
      <TLD val="nl" count="1"/>
      <TLD val="museum" count="1"/>
      <TLD val="ie" count="1"/>
    </Countries>
...

With PSL list configured: 

<array name="countries" dimensions="2" start_time="1683724140" stop_time="1683724200">
  <dimension number="1" type="Countries"/>
  <dimension number="2" type="TLD"/>
  <data>
    <Countries val="BA">
      <TLD val="com" count="113"/>
      <TLD val="de" count="49"/>
      <TLD val="net" count="36"/>
      <TLD val="org" count="29"/>
      <TLD val="int" count="27"/>
      <TLD val="at" count="14"/>
      <TLD val="no" count="7"/>
      <TLD val="eu" count="4"/>
      <TLD val="hr" count="3"/>
      <TLD val="ch" count="3"/>
      <TLD val="dev" count="3"/>
      <TLD val="cz" count="2"/>
      <TLD val="tv" count="2"/>
      <TLD val="gl" count="2"/>
      <TLD val="sexy" count="2"/>
      <TLD val="online" count="2"/>
      <TLD val="io" count="2"/>
      <TLD val="co.at" count="1"/>
      <TLD val="sk" count="1"/>
      <TLD val="ba" count="1"/>
      <TLD val="porgy.at" count="1"/>
      <TLD val="frischeis.ba" count="1"/>
      <TLD val="hudportalen.no" count="1"/>
      <TLD val="museum" count="1"/>
      <TLD val="name" count="1"/>
      <TLD val="global" count="1"/>
      <TLD val="network" count="1"/>
    </Countries>
...

So, suddenly there appear 2nd level domains, with and without public suffix, in the TLDs:

      <TLD val="co.at" count="1"/>

That is not wrong, alhtough strictly it is not a TLD. So that may be fine.

      <TLD val="frischeis.ba" count="1"/>

That is clearly wrong, the string frischeis is not configured in our suffix list.

Config:

local_address ......
run_dir "/var/spool/dsc/run/regdns";
minfree_bytes 5000000;

pid_file "/var/run/regdns.dsc.pid";

statistics_interval 60;

bpf_program "(net ........";

interface eth0;
interface gre13;
interface gre14;

dataset qtype dns All:null Qtype:qtype queries-only;
dataset rcode dns All:null Rcode:rcode replies-only;
dataset opcode dns All:null Opcode:opcode queries-only;
dataset rcode_vs_replylen dns Rcode:rcode ReplyLen:msglen replies-only;
dataset client_subnet dns All:null ClientSubnet:client_subnet queries-only max-cells=200;
dataset qtype_vs_qnamelen dns Qtype:qtype QnameLen:qnamelen queries-only;
dataset qtype_vs_tld dns Qtype:qtype TLD:tld queries-only,popular-qtypes max-cells=200;
dataset certain_qnames_vs_qtype dns CertainQnames:certain_qnames Qtype:qtype queries-only;
dataset client_subnet2 dns Class:query_classification ClientSubnet:client_subnet queries-only max-cells=200;
dataset client_addr_vs_rcode dns Rcode:rcode ClientAddr:client replies-only max-cells=50;
dataset chaos_types_and_names dns Qtype:qtype Qname:qname chaos-class,queries-only;
dataset idn_qname dns All:null IDNQname:idn_qname queries-only;
dataset edns_version dns All:null EDNSVersion:edns_version queries-only;
dataset edns_bufsiz dns All:null EDNSBufSiz:edns_bufsiz queries-only;
dataset do_bit dns All:null D0:do_bit queries-only;
dataset rd_bit dns All:null RD:rd_bit queries-only;
dataset idn_vs_tld dns All:null TLD:tld queries-only,idn-only;
dataset ipv6_rsn_abusers dns All:null ClientAddr:client queries-only,aaaa-or-a6-only,root-servers-net-only max-cells=50;
dataset transport_vs_qtype dns Transport:transport Qtype:qtype queries-only;
dataset client_port_range dns All:null PortRange:dns_sport_range queries-only;
dataset second_ld_vs_rcode dns Rcode:rcode SecondLD:second_ld replies-only max-cells=50;
dataset third_ld_vs_rcode dns Rcode:rcode ThirdLD:third_ld replies-only max-cells=50;
dataset direction_vs_ipproto ip Direction:ip_direction IPProto:ip_proto any;
dataset dns_ip_version_vs_qtype dns IPVersion:dns_ip_version Qtype:qtype queries-only;
dataset countries dns Countries:country TLD:tld queries-only;
dataset countries_all dns All:null Countries:country queries-only;
dataset asn_all dns IPVersion:dns_ip_version ASN:asn queries-only max-cells=200;
dataset nxdomains dns All:null Qname:qname nxdomains-only max-cells=200;
dataset transport_queries dns All:null Transport:transport queries-only;
dataset transport_replies dns All:null Transport:transport replies-only;
dataset resptime dns All:null ResponseTime:response_time;
#dataset resptime_per_qtype dns Qtype:qtype ResponseTime:response_time;
maxminddb_asn "/var/lib/GeoIP/GeoLite2-ASN.mmdb";
maxminddb_country "/var/lib/GeoIP/GeoLite2-Country.mmdb";
asn_indexer_backend maxminddb;
country_indexer_backend maxminddb;
no_wait_interval;
dump_reports_on_exit;
tld_list /etc/dsc-statistics/tldlist.txt;

tldlist.txt

jelu commented 1 year ago

The tld_list option changes how TLDs are interpret throughout all of DSC, including tld indexer. It's in the man-page:

This option changes what DSC considers a TLD (similar to Public Suffix List) and affects any indexers that gathers statistics on TLDs

The co.at is correct because of that.

$ grep co.at tldlist.txt 
co.at

The frischeis.ba is not, but I am unable to replicate the result (see below). Can you create a reproducible setup/PCAP?

$ tcpdump -nvr test.pcap 
reading from file test.pcap, link-type LINUX_SLL (Linux cooked v1)
12:57:03.682234 IP (tos 0x0, ttl 64, id 40525, offset 0, flags [none], proto UDP (17), length 85)
    172.17.0.6.33676 > 172.17.0.1.53: 6853+ [1au] A? www.frischeis.ba. (57)
12:57:03.682305 IP (tos 0x0, ttl 64, id 19473, offset 0, flags [none], proto UDP (17), length 103)
    172.17.0.1.53 > 172.17.0.6.33676: 6853 2/0/1 www.frischeis.ba. CNAME frischeis.ba., frischeis.ba. A 13.248.176.187 (75)

$ src/dsc -dddd dsc.conf 
adding local address 127.0.0.1
adding local address ::1
setting current directory to .
minfree_bytes 5000000
Opening interface test.pcap
creating dataset countries
country_indexer using MaxMind DB backend
Maxmind ASN database /home/jerry/GeoIP/GeoLite2-Country.mmdb
loaded TLD list from tldlist.txt
country_index: Sucessfully initialized MaxMind Country
asn_index: No database loaded for GeoIP ASN
Running
     172.17.0.6:33676   UDP QT=1    QC=1    len=57  qname=www.frischeis.ba  tld=ba  opcode=0    rcode=0 malformed=0 qr=0    rd=1
country_index: country code: ?4
     172.17.0.6:33676   UDP QT=1    QC=1    len=75  qname=www.frischeis.ba  tld=ba  opcode=0    rcode=0 malformed=0 qr=1    rd=1
writing to 1683802623.dscdata.xml.XXXB1AZ3g
renaming to 1683802623.dscdata.xml

$ cat 1683802623.dscdata.xml 
<dscdata>
<array name="pcap_stats" dimensions="2" start_time="1683802623" stop_time="1683802623">
  <dimension number="1" type="ifname"/>
  <dimension number="2" type="pcap_stat"/>
  <data>
    <ifname val="test.pcap">
      <pcap_stat val="pkts_captured" count="2"/>
    </ifname>
  </data>
</array>
<array name="countries" dimensions="2" start_time="1683802623" stop_time="1683802623">
  <dimension number="1" type="Countries"/>
  <dimension number="2" type="TLD"/>
  <data>
    <Countries val="PzQ=" base64="1">
      <TLD val="ba" count="1"/>
    </Countries>
  </data>
</array>
</dscdata>

$ cat dsc.conf | grep -v -e '^#' -e '^$'
local_address 127.0.0.1;
local_address ::1;
run_dir ".";
minfree_bytes 5000000;
interface test.pcap;
dataset countries dns Countries:country TLD:tld queries-only;
country_indexer_backend maxminddb;
maxminddb_country "/home/jerry/GeoIP/GeoLite2-Country.mmdb";
tld_list tldlist.txt;
Philambdo commented 1 year ago

Hey Jerry,

i work with Klaus in the same team. Thanks for your response.

I tried to reproduce the error as good as i can - you find an pcap in the attachment. Things i noticed: The problem only happens with .at, .org looks fine. I'm unsure if thirdld should look the way it does Also everything seems to work better if i use www.blabla.at / www.blabla.org as query.

Facts:

The dsc version we use is the following:

ii  dsc                                   2.14.0-1~jammy+1                        amd64        DNS Statistics Collector

I did a dig vs my localhost pdns

dig -t NS blabla.at @127.0.0.1
dig -t NS blabla.org @127.0.0.1

I used the following config on our dsc:

local_address 127.0.0.1;
run_dir "/var/run/dsc";
minfree_bytes 5000000;
pid_file "/var/run/dsc.pid";
statistics_interval 60;
interface lo;

dataset qtype_vs_tld dns Qtype:qtype TLD:tld queries-only,popular-qtypes max-cells=200;
dataset second_ld_vs_rcode dns Rcode:rcode SecondLD:second_ld replies-only max-cells=50;
dataset third_ld_vs_rcode dns Rcode:rcode ThirdLD:third_ld replies-only max-cells=50;
dataset countries dns Countries:country TLD:tld queries-only;
maxminddb_asn "/etc/dsc/GeoLite2-ASN.mmdb";
maxminddb_country "/etc/dsc/GeoLite2-Country.mmdb";
asn_indexer_backend maxminddb;
country_indexer_backend maxminddb;
no_wait_interval;
dump_reports_on_exit;

tld_list /etc/dsc/tld.list;

The resulting xml:

<dscdata>
<array name="pcap_stats" dimensions="2" start_time="1683879747" stop_time="1683879752">
  <dimension number="1" type="ifname"/>
  <dimension number="2" type="pcap_stat"/>
  <data>
    <ifname val="lo">
      <pcap_stat val="filter_received" count="8"/>
      <pcap_stat val="pkts_captured" count="4"/>
    </ifname>
  </data>
</array>
<array name="countries" dimensions="2" start_time="1683879747" stop_time="1683879752">
  <dimension number="1" type="Countries"/>
  <dimension number="2" type="TLD"/>
  <data>
    <Countries val="PzQ=" base64="1">
      <TLD val="org" count="1"/>
      <TLD val="blabla.at" count="1"/>
    </Countries>
  </data>
</array>
<array name="third_ld_vs_rcode" dimensions="2" start_time="1683879747" stop_time="1683879752">
  <dimension number="1" type="Rcode"/>
  <dimension number="2" type="ThirdLD"/>
  <data>
    <Rcode val="5">
      <ThirdLD val="blabla.org" count="1"/>
      <ThirdLD val="blabla.at" count="1"/>
    </Rcode>
  </data>
</array>
<array name="second_ld_vs_rcode" dimensions="2" start_time="1683879747" stop_time="1683879752">
  <dimension number="1" type="Rcode"/>
  <dimension number="2" type="SecondLD"/>
  <data>
    <Rcode val="5">
      <SecondLD val="blabla.org" count="1"/>
      <SecondLD val="blabla.at" count="1"/>
    </Rcode>
  </data>
</array>
<array name="qtype_vs_tld" dimensions="2" start_time="1683879747" stop_time="1683879752">
  <dimension number="1" type="Qtype"/>
  <dimension number="2" type="TLD"/>
  <data>
    <Qtype val="2">
      <TLD val="org" count="1"/>
      <TLD val="blabla.at" count="1"/>
    </Qtype>
  </data>
</array>
</dscdata>

Here the pcap dump created with:

tcpdump -nntti lo port 53 and host 127.0.0.1 -s0 -w dump.pcap

dump.tar.gz

jelu commented 1 year ago

@Philambdo Thanks, was able to replicate and fix problem.

Can you verify yourself using compiled develop branch or pre-release packages?

I'm afraid I won't have time to do a release until after RIPE86.

Philambdo commented 1 year ago

Hey Jerry,

wow that was quick.

I cloned the current master, and compiled it. It seems everything is working now as expected.

I will do the final testing with the next release on our production servers.

Thanks again for the fast fix.

Greetings, Florian