This ticket isn't yet asking for a specific resolution, but I've opened this to gather opinions on what (if anything) we should do about domains which return a DNS response of NXDOMAIN. Generally these seem to be domains for which registration has expired.
These domains still exist quite prevalently in the database (which makes sense since we could only really detect them by continually re-checking all of the unique URLs we've tested so far.
But this also means that when creating a new list, there's often a lot of domains which get pulled into it which no longer exist and it's arguable that they have no place in the list.
But, as a counterpoint, these are domains which did exist at one point (and we have metadata for them), and which we potentially tested, had a report for, and subsequently re-tested as unblocked. Is the fact that the domain registration has since lapsed damaging to our statistics?
Should we be periodically testing lists and purging all NXDOMAIN responses from them? or should we be leaving those entries in lists as examples of previously blocked-then-unblocked sites?
We briefly discussed this @JimKillock so I'm just looking for a general discussion in advance of the report publication.
This ticket isn't yet asking for a specific resolution, but I've opened this to gather opinions on what (if anything) we should do about domains which return a DNS response of
NXDOMAIN. Generally these seem to be domains for which registration has expired.These domains still exist quite prevalently in the database (which makes sense since we could only really detect them by continually re-checking all of the unique URLs we've tested so far.
But this also means that when creating a new list, there's often a lot of domains which get pulled into it which no longer exist and it's arguable that they have no place in the list.
But, as a counterpoint, these are domains which did exist at one point (and we have metadata for them), and which we potentially tested, had a report for, and subsequently re-tested as unblocked. Is the fact that the domain registration has since lapsed damaging to our statistics?
Should we be periodically testing lists and purging all
NXDOMAINresponses from them? or should we be leaving those entries in lists as examples of previously blocked-then-unblocked sites?We briefly discussed this @JimKillock so I'm just looking for a general discussion in advance of the report publication.