Tiny issues in data result in "500 Internal Server Error" in API, while the server responds correctly via DNS protocol

martenlehmann commented 1 year ago

Program: Authoritative
Issue type: Bug report

Short description

I noticed in several cases, that PowerDNS raises an exception in the log and returns 500 Internal Server Error in the API on records that it serves just fine via DNS queries. These exceptions happen at least on a zone info and on zone export.

This should never happen, as it's not obvious which other situations and data formats might result in similar exceptions, so these exceptions can't be prevented for sure (there might be lots of unexpected input despite some validation). However, 500 errors in an API prevent debugging the problem and resuming normal operations while doing so. On top of that, the log message uses the term "should", which means it gives a recommendation that is optional. But as the code makes it actually mandatory and raises exceptions otherwise, the handling needs to be considered broken.

Environment

Operating system: Ubuntu 22.04.2 LTS
Software version: 4.7.3-1pdns.jammy
Software source: PowerDNS repository

Steps to reproduce

Insert a TXT record with a minor issue (not using quotes for your content for example, which is redundant anyway when using a database backend)
Try to retrieve the zone via API and get a 500 Internal Server Error
Find a log message like HTTP ISE for "/api/v1/servers/localhost/zones/example.com.": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'v=spf1 include:spf.example.com ?all'
Resolve the TXT record via DNS just fine without errors and without log messages for comparison

Expected behaviour

Log the problem as a warning (the underlying issue should be for debate anyway), yet have all interfaces operating smoothly.

Actual behaviour

500 Internal Server Error responses, which break automated systems from operating and applying / verifying changes.

Habbie commented 1 year ago

Did you notice this issue with types other than TXT?

martenlehmann commented 1 year ago

I traced down the exception regarding the missing quotes to RecordTextException() in rcpgenerator.cc. There are other cases in which this exception is raised, but none that are relevant to our set of record types.

There is another case in which an exception is raised on unknown records that don't start with \# (Unknown record was stored incorrectly, first part should be '\#', got '#'). It's raised via MOADNSException() in dnsparser.cc.

I'm going to compare the behavior on these two.

When querying a TXT record without quotes:

Response: Just normal, PowerDNS adds leading and trailing quotes on the fly
Log: Nothing

When querying an unknown record without leading \#:

Response: SERVFAIL

Log:

Exception building answer packet for type65280.example.com/TYPE65280 (Unknown record was stored incorrectly, first part should be '\#', got '#') sending out servfail

When requesting AXFR for a zone with a TXT record without quotes:

Response: Just normal, full zone, PowerDNS adds leading and trailing quotes on the fly
Log: Nothing

When requesting AXFR for a zone with an unknown record without leading \#:

Response: communications error to <127.0.0.1>#53: end of file after SOA record (and additional RRSIG for SOA with DNSSEC enabled)

Log (twice, as dig seems to attempt a second query automatically):

AXFR-out zone 'example.com', client '127.0.0.1', transfer initiated
TCP Connection Thread died because of STL error, cycling backend: Unknown record was stored incorrectly, first part should be '\#', got '#'
AXFR-out zone 'example.com', client '127.0.0.1', transfer initiated
TCP server is without backend connections in doAXFR, launching
TCP Connection Thread died because of STL error, cycling backend: Unknown record was stored incorrectly, first part should be '\#', got '#'

When requesting /export via API for a zone with a TXT record without quotes:

Response: 500 Internal Server Error

Log:

HTTP ISE for "/api/v1/servers/localhost/zones/example.com./export": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'raw data'

When requesting /export via API for a zone with an unknown record without leading \#:

Response: 500 Internal Server Error

Log:

HTTP ISE for "/api/v1/servers/localhost/zones/example.com./export": STL Exception: Unknown record was stored incorrectly, first part should be '\#', got '#'

When executing pdnsutil check-zone on a zone with a TXT record without quotes: No issues.

When executing pdnsutil check-zone on a zone with an unknown record without leading \#:

[Error] Following record had a problem: "type65280.example.com IN TYPE65280 # 13 076578616d706c65036e657400"
[Error] Error was: Unknown record was stored incorrectly, first part should be '\#', got '#'

Interestingly a zone lacking NS records receives a message of severity "Error" as well, yet no exception or Internal Server Error:

[Error] No NS record at zone apex in zone 'example.com'

Also interesting: There doesn't seem to be an endpoint in the API to provide access to the results of pdnsutil check-zone.

So what worries me is:

These two exceptions have different effects on operation and the handling seems inconsistent.
There is no way for to retrieve the error messages / exceptions via the API.
While these errors seem to be only two affecting us right now, this might change in the future, just like unknown record without leading \# hasn't been an issue in previous releases when PowerDNS was more tolerant. PowerDNS already shows warnings regarding issues with trailing dots. So if these warnings are raised to errors and exceptions with a future release, that's like a damocles sword, as this would break chronological processing of jobs in a queue.

ckbaker10 commented 1 year ago

Apr 3 16:12:49 ns1 pdns_server[1010]: [webserver] d8ed8dc6-66bb-482c-85bd-56fb6b4f728d HTTP ISE for "/api/v1/servers/localhost/zones/custom.zone.": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'unescapeddnsstring'

This error cannot be identified by using pdnsutil check-all-zones

I came across it writing sanity checks for my dns setup, by querying all zones via the http api I happened to encounter a 500 internal error

Producing line: https://github.com/PowerDNS/pdns/blob/master/pdns/rcpgenerator.cc#L592

For me, there was no fix beside direct sql editing

pdnsutil edit-zone refuses to open affected zones

dnsdist/unknown,now 1.7.3-1pdns.jammy amd64 [installed]
pdns-backend-mysql/unknown,now 4.7.3-1pdns.jammy amd64 [installed]
pdns-recursor/now 4.8.3-1pdns.jammy amd64 [installed,upgradable to: 4.8.4-1pdns.jammy]
pdns-server/unknown,now 4.7.3-1pdns.jammy amd64 [installed]

zones were created using zone2sql migrating from a bind9 text configuration, data source was configured this way. Would be nice if zone2sql attempted to fix.

PowerDNS / pdns