Rate limit whois requests?

[mrnan00]

Is there a rate limit for the whois requests necessary?

I got a lot of false positives for the signature "_generic_cname_found_but_unregistered". I tracked down the issue to the def is_registered(self): method in domain.py: https://github.com/punk-security/dnsReaper/blob/446ce7192bf77b22ab75a3c82a366e661af97c27/domain.py#L80

If I make a loop over the whois part, I will eventually hit the point where whois.whois(self.domain)["registrar"] is None, even if the domain is registered. In the whois.text field I find the error message: "'The number of requests per client per time interval is restricted. You have exceeded this limit. Please wait a moment and try again." There is no exception thrown from whois to catch this. Maybe add a backoff it there is an error message in the whois.text field?

[SimonGurney]

Thanks for the report and for digging into this for us.

We only ever check whois data for unresolvable CNAME records which are no more than two parts, i.e. foo.com would be checked, but bar.foo.com would not be. In our testing so far, this has meant that our testing has never flagged this issue.

This is a thorny issue, as the text is returned from the whois server and could be different for each tld. We were really hoping to get an exception from the whois library itself.

For now, you can disable that signature.

--exclude-signature _generic_cname_found_but_unregistered

Which TLD were you testing against? .com ?

[mrnan00]

I discovered it too, that the text is different for each tld. The issue was discovered with .ch domains. And with .es TLD I discovered, that the whois services denies access.

.es response: Conditions of use for the whois service via port 43 for .es domains Access will only be enabled for IP addresses authorised by Red.es. A maximum of one IP address per user/organisation is permitted. Red.es accepts no responsibility whatsoever for the availability of access to WHOIS, which may be suspended at any time and without prior warning at the discretion of the public entity.

I think whois is not very reliable for certain tld's but I have no better idea to solve this issue. Whois is the best we have at the moment.

One idea is to look out for NXDOMAIN answer, but this does not necessarily mean the domain is not already registered. But maybe it is more robust than whois. But I have to admit that I am not quite up to speed with DNS.

[SimonGurney]

Ok I have a fix for this. The library we use will raise an error for a confirmed no match and one other random scenario. We can check check for this other scenario and then use the exception itself to confirm its unregistered

    def is_registered(self):
        try:
            whois.whois(self.domain)
            return True
        except whois.parser.PywhoisError as e:
            if e.args[0] == "No whois server is known for this kind of object.":
                # This is the only case of a potentially registered domain
                # triggering a PywhoisError
                # https://github.com/richardpenman/whois/blob/56dc7e41d134e6d4343ad80a48533681bd887ff2/whois/parser.py#L201
                return True
            return False
        except Exception:
            return True

[SimonGurney]

We havent added rate limiting as we have a parallel thread control switch anyway, but the false positives are now gone!

Thanks for the bug :)

[mrnan00]

Thanks for fixing it so quickly!