threatexpress / domainhunter

Checks expired domains for categorization/reputation and Archive.org history to determine good candidates for phishing and C2 domain names
http://threatexpress.com
BSD 3-Clause "New" or "Revised" License
1.52k stars 288 forks source link

MXToolbox reputation checking is broken #21

Open andrewchiles opened 4 years ago

andrewchiles commented 4 years ago

The service has removed the original endpoints used to query SPAM and Google Safe Browsing lists. There is a new API that returns JSON objects and is heavily reliant on JS to format the requests correctly. Any malformed request results in an IP block that requires CAPTCHA completion to remove.

All HTTP GETs

  1. https://mxtoolbox.com/domain/apples.com/ -> Initial request to the service
  2. https://mxtoolbox.com/api/v1/user -> Returns JSON that is subsequently used as cookie parameter and also an HTTP header in the GET. Need to parse and manually update requests session cookie jar and create custom HTTP headers
  3. https://mxtoolbox.com/api/v1/lookup/blacklist/apples.com -> Returns JSON with blacklist lookup results. All we really need to check is presence of records in "Failed" or "Warnings"
    {
    "UID": null,
    "ArgumentType": "hostname",
    "Command": "blacklist",
    "IsTransitioned": false,
    "CommandArgument": "nwk-aaemail-lapp01.apple.com",
    "TimeRecorded": "2020-02-05T15:54:56.7293086-06:00",
    "ReportingNameServer": null,
    "TimeToComplete": "328",
    "RelatedIP": "17.151.62.66",
    "ResourceRecordType": 0,
    "IsEmptySubDomain": false,
    "IsEndpoint": true,
    "HasSubscriptions": false,
    "AlertgroupSubscriptionId": null,
    "Failed": [],
    "Warnings": [],
    "Passed": [
    {
      "ID": 333,
      "Name": "BSB Domain",
      "Url": "https://mxtoolbox.com/Problem/blacklist/BSB-Domain?page=prob_blacklist&showlogin=1&hidetoc=1&action=blacklist:nwk-aaemail-lapp01.apple.com",
      "PublicDescription": null,
      "BlacklistResponseTime": "0",
      "IsExcludedByUser": false
    },
    SNIP
    ],
    "Errors": [],
    "IsError": false,
    "Information": [
    {
      "DNS Resolution": "nwk-aaemail-lapp01.apple.com was resolved to 17.151.62.66."
    }
    ],
    "MultiInformation": [],
    "IsBruteForce": false,
    "Transcript": [
    {
      "Transcript": "DNS - Load Balancers\r\nLookupServer 328ms\r\n"
    }
    ],
    "MxRep": 100,
    "EmailServiceProvider": null,
    "DnsServiceProvider": null,
    "DnsServiceProviderIdentifier": null,
    "RelatedLookups": [
    {
      "Name": "dns lookup",
      "URL": "https://mxtoolbox.com/api/v1/lookup/a/nwk-aaemail-lapp01.apple.com",
      "Command": "a",
      "CommandArgument": "nwk-aaemail-lapp01.apple.com"
    },
    {
      "Name": "smtp diag",
      "URL": "https://mxtoolbox.com/api/v1/lookup/smtp/nwk-aaemail-lapp01.apple.com",
      "Command": "smtp",
      "CommandArgument": "nwk-aaemail-lapp01.apple.com"
    },
    {
      "Name": "http test",
      "URL": "https://mxtoolbox.com/api/v1/lookup/http/nwk-aaemail-lapp01.apple.com",
      "Command": "http",
      "CommandArgument": "nwk-aaemail-lapp01.apple.com"
    }
    ]
    }
acole76 commented 3 years ago

All we really need to check is presence of records in "Failed" or "Warnings"

it looks like checkMXToolbox() is checking both Google safe browsing and Phish tank. What would be the expected return value for checkMXToolbox() if there were records in the failed or warning?