The OpenDNS Investigate documentation states that in case a 404 is returned the resource does not exists, it is useless to retry those requests. An optional parameter was added to the MultiRequest HTTP handler to allow that.
While I was at it, I bumped the version of pre-commit hooks (sorry for the large diff). Significant changes are at threat_intel/opendns.py:64 and threat_intel/util/http.py:143,349
Testing:
>>> from threat_intel.opendns import InvestigateApi
>>> inv = InvestigateApi('<redacted>')
>>> domains = ['fyeipf.com', 'boomskala.ooo', 'google.com', 'yeipsb.com']
>>> dir(inv)
['BASE_URL', 'MAX_DOMAINS_IN_POST', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_cache', '_multi_get', '_multi_post', '_requests', '_to_url', '_to_urls', 'categorization', 'cooccurrences', 'dns_rr', 'domain_score', 'domain_tag', 'latest_malicious', 'related_domains', 'risk_score', 'rr_history', 'sample', 'search', 'security', 'whois_domains', 'whois_domains_history', 'whois_emails', 'whois_nameservers']
>>> inv.whois_domains(domains)
WARNING:root:Request to https://investigate.api.opendns.com/whois/fyeipf.com failed with status code 404, dropping.
WARNING:root:Request to https://investigate.api.opendns.com/whois/boomskala.ooo failed with status code 404, dropping.
WARNING:root:Request to https://investigate.api.opendns.com/whois/yeipsb.com failed with status code 404, dropping.
{'fyeipf.com': None, 'google.com': {....}, 'boomskala.ooo': None, 'yeipsb.com': None}
The OpenDNS Investigate documentation states that in case a 404 is returned the resource does not exists, it is useless to retry those requests. An optional parameter was added to the
MultiRequest
HTTP handler to allow that. While I was at it, I bumped the version of pre-commit hooks (sorry for the large diff). Significant changes are atthreat_intel/opendns.py:64
andthreat_intel/util/http.py:143,349
Testing: