search

PowerDNS / pdns

PowerDNS Authoritative, PowerDNS Recursor, dnsdist
https://www.powerdns.com/
GNU General Public License v2.0
3.61k stars 900 forks source link

Add User-Agent: logging data to dnstap #9102

Open johnhtodd opened 4 years ago

johnhtodd commented 4 years ago

Short description

Create an addition to dnstap (and/or pdns protobuf format) that includes the User-Agent: string for remote logging and analysis.

Usecase

There are now many DoH clients, and more arriving every day. While some may not include User-Agent, some will. It is a useful data point to understand what is the volume of client requests coming from sources that include User-Agent, and what those User-Agent strings are set to if they are complete. Including this in the dnstap data would allow for external analysis.

Description

Optional inclusion of User-Agent: string contents from DoH requests in dnstap messages. This may be a one-to-many model, since the DoH socket will remain open with many requests. The reported User-Agent string should be included in each dnstap logging event. The string should be limited to some reasonable length before insertion to prevent logging overload.

It will probably be the case that some key model needs to be implemented in the dnstap extension field, as User-Agent may only be the first of several different components of extension in the data stream. Habbie suggested json as an encoding model.

Reference: dnstap extension fields: https://github.com/dnstap/dnstap.pb/blob/master/dnstap.proto#L40

rgacogne commented 4 years ago

Habbie suggested json as an encoding model.

JSON would work quite well, I'm wondering if a nested protobuf message would not make more sense in term of speed and overall size.

johnhtodd commented 4 years ago

I don't see any attachment a particular model. TLV, protobuf, json - I leave that to those who understand the performance bottlenecks better.

johnhtodd commented 1 month ago

I would be fine just reducing this down to adding the field to a "Meta" field in just the PowerDNS protobuf format structures, since there is no place in the DNSTAP structures that it would make sense. Even whittling it down further, I could even survive if this was just available somewhere to Lua and I can cram it into the extra field or log it or something else. (is it already available?)

rgacogne commented 1 month ago

The HTTP User-Agent is available to Lua via DNSQuestion:getHTTPHeaders1, and the DnstapLogAction callback function 2 can be used to set the extra field via DnstapMessage:setExtra3.