feat: add SPF PTR support (since Yahoo still uses)

[titanism]

We're writing as we discovered an edge case where vacation auto-responders from Yahoo would normally have passing SPF and DKIM (thus a passing DMARC), but when using mailauth, not only do we get a negative DMARC result, we also get a negative SPF and DKIM result too. Gmail for example reports these vacation response messages from Yahoo as having passing DMARC, DKIM, and SPF.

We've included a complete test for you as well so you can see the result.

mailauth --verbose --client-ip 66.163.189.147 --sender forwardemailtest@yahoo.com --helo sonic314-21.consmr.mail.ne1.yahoo.com --mta mx1.forwardemail.net test.txt
Reading email message from test.txt
{
  "dkim": {
    "headerFrom": [
      "forwardemailtest@yahoo.com"
    ],
    "envelopeFrom": "forwardemailtest@yahoo.com",
    "results": [
      {
        "signingDomain": "yahoo.com",
        "selector": "s2048",
        "signature": "U6xSIk/z+xTpR2vRhfKVn9ONotJWf2WTvroxSg5/kCtQ+zxcvUdCaH80Py+Cz7TtieCy77/kRqFzpJTp1R1OJf+m9TeS2DUaijVM8RKdbmHmZhRY6/YwKyJdhj9+9z6y7jVcxqS+7+ib+wqipOKE5Q6p87zLlXfOXgYsFh/Bott86PFmf5sKhf1C/LkmPreJE4ep21AU7NM4zaTLhY8fS2fdKDDXq2cyaE8LKu9CkWCcUaMqmFJ11GQetDZvCD5nBIR89GvM8IAWmPWWFGSD7c1AJcOC51+1ILcfsg8IqPhLUPpDPsRbT2Z31LI9clxlPvaI6yRanTnj0Cf3LJ5UIQ==",
        "algo": "rsa-sha256",
        "format": "relaxed/relaxed",
        "bodyHash": "mD4IKcP0SaNBDF959LqTWtanj/53dx+DHlUfLpHedO4=",
        "bodyHashExpecting": "mD4IKcP0SaNBDF959LqTWtanj/53dx+DHlUfLpHedO4=",
        "signingHeaders": {
          "keys": "Date: From: To: Subject",
          "headers": [
            "Date: Fri, 29 Jul 2022 19:22:11 +0000 (UTC)",
            "From: Forwared Email <forwardemailtest@yahoo.com>",
            "To: <pm_bounces@pm-bounces.forwardemail.net>",
            "Subject: Auto Response: Attempt to show failure"
          ]
        },
        "status": {
          "result": "fail",
          "header": {
            "i": "@yahoo.com",
            "s": "s2048",
            "a": "rsa-sha256",
            "b": "U6xSIk/z"
          },
          "aligned": "yahoo.com"
        },
        "canonBodyLength": 371,
        "publicKey": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuoWufgbWw58MczUGbMv1\n76RaxdZGOMkQmn8OOJ/HGoQ6dalSMWiLaj8IMcHC1cubJx2gziAPQHVPtFYayyLA\n4ayJUSNk10/uqfByiU8qiPCE4JSFrpxflhMIKV4bt+g1uHw7wLzguCf4YAoR6XxU\nKRsAoHuoF7M+v6bMZ/X1G+viWHkBl4UfgJQ6O8F1ckKKoZ5KqUkJH5pDaqbgs+F3\nPpyiAUQfB6EEzOA1KMPRWJGpzgPtKoukDcQuKUw9GAul7kSIyEcizqrbaUKNLGAm\nz0elkqRnzIsVpz6jdT1/YV5Ri6YUOQ5sN5bqNzZ8TxoQlkbVRy6eKOjUnoSSTmSA\nhwIDAQAB\n-----END PUBLIC KEY-----",
        "modulusLength": 2048,
        "rr": "k=rsa;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuoWufgbWw58MczUGbMv176RaxdZGOMkQmn8OOJ/HGoQ6dalSMWiLaj8IMcHC1cubJx2gziAPQHVPtFYayyLA4ayJUSNk10/uqfByiU8qiPCE4JSFrpxflhMIKV4bt+g1uHw7wLzguCf4YAoR6XxUKRsAoHuoF7M+v6bMZ/X1G+viWHkBl4UfgJQ6O8F1ckKKoZ5KqUkJH5pDaqbgs+F3PpyiAUQfB6EEzOA1KMPRWJGpzgPtKoukDcQuKUw9GAul7kSIyEcizqrbaUKNLGAmz0elkqRnzIsVpz6jdT1/YV5Ri6YUOQ5sN5bqNzZ8TxoQlkbVRy6eKOjUnoSSTmSAhwIDAQAB;",
        "info": "dkim=fail header.i=@yahoo.com header.s=s2048 header.a=rsa-sha256 header.b=\"U6xSIk/z\""
      }
    ]
  },
  "spf": {
    "domain": "yahoo.com",
    "client-ip": "66.163.189.147",
    "helo": "sonic314-21.consmr.mail.ne1.yahoo.com",
    "envelope-from": "forwardemailtest@yahoo.com",
    "rr": "v=spf1 redirect=_spf.mail.yahoo.com",
    "status": {
      "result": "neutral",
      "comment": "mx1.forwardemail.net: 66.163.189.147 is neither permitted nor denied by domain of forwardemailtest@yahoo.com",
      "smtp": {
        "mailfrom": "forwardemailtest@yahoo.com",
        "helo": "sonic314-21.consmr.mail.ne1.yahoo.com"
      }
    },
    "header": "Received-SPF: neutral (mx1.forwardemail.net: 66.163.189.147 is neither permitted nor denied by domain of forwardemailtest@yahoo.com) client-ip=66.163.189.147;",
    "info": "spf=neutral (mx1.forwardemail.net: 66.163.189.147 is neither permitted nor denied by domain of forwardemailtest@yahoo.com) smtp.mailfrom=forwardemailtest@yahoo.com smtp.helo=sonic314-21.consmr.mail.ne1.yahoo.com",
    "lookups": {
      "limit": 50,
      "count": 2
    }
  },
  "dmarc": {
    "status": {
      "result": "fail",
      "comment": "p=REJECT arc=none",
      "header": {
        "from": "yahoo.com",
        "d": "yahoo.com"
      }
    },
    "domain": "yahoo.com",
    "policy": "reject",
    "p": "reject",
    "sp": "reject",
    "pct": 100,
    "rr": "v=DMARC1; p=reject; pct=100; rua=mailto:d@rua.agari.com; ruf=mailto:d@ruf.agari.com;",
    "alignment": {
      "spf": {
        "result": false,
        "strict": false
      },
      "dkim": {
        "result": false,
        "strict": false
      }
    },
    "info": "dmarc=fail (p=REJECT arc=none) header.from=yahoo.com header.d=yahoo.com"
  },
  "arc": {
    "status": {
      "result": "none"
    },
    "i": 0,
    "authResults": "mx1.forwardemail.net;\r\n dkim=fail header.i=@yahoo.com header.s=s2048 header.a=rsa-sha256 header.b=\"U6xSIk/z\";\r\n spf=neutral (mx1.forwardemail.net: 66.163.189.147 is neither permitted nor denied by domain of forwardemailtest@yahoo.com)\r\n smtp.mailfrom=forwardemailtest@yahoo.com smtp.helo=sonic314-21.consmr.mail.ne1.yahoo.com;\r\n dmarc=fail (p=REJECT arc=none) header.from=yahoo.com header.d=yahoo.com;\r\n bimi=skipped (message failed DMARC)"
  },
  "bimi": {
    "status": {
      "header": {},
      "result": "skipped",
      "comment": "message failed DMARC"
    },
    "info": "bimi=skipped (message failed DMARC)"
  },
  "receivedChain": [
    {
      "from": {
        "value": "sonic.gate.mail.ne1.yahoo.com"
      },
      "by": {
        "value": "sonic314.consmr.mail.ne1.yahoo.com"
      },
      "with": {
        "value": "HTTP"
      },
      "timestamp": "Fri, 29 Jul 2022 19:22:11 +0000",
      "full": "Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.ne1.yahoo.com with HTTP; Fri, 29 Jul 2022 19:22:11 +0000"
    }
  ],
  "headers": "Received-SPF: neutral (mx1.forwardemail.net: 66.163.189.147 is neither permitted nor denied by domain of forwardemailtest@yahoo.com) client-ip=66.163.189.147;\r\nAuthentication-Results: mx1.forwardemail.net;\r\n dkim=fail header.i=@yahoo.com header.s=s2048 header.a=rsa-sha256 header.b=\"U6xSIk/z\";\r\n spf=neutral (mx1.forwardemail.net: 66.163.189.147 is neither permitted nor denied by domain of forwardemailtest@yahoo.com)\r\n smtp.mailfrom=forwardemailtest@yahoo.com smtp.helo=sonic314-21.consmr.mail.ne1.yahoo.com;\r\n dmarc=fail (p=REJECT arc=none) header.from=yahoo.com header.d=yahoo.com;\r\n bimi=skipped (message failed DMARC)\r\n"
}

Contents of test-new.eml are attached: test.txt

[titanism]

We also discovered another bug at https://github.com/postalsys/mailauth/blob/bcf569fd9f351c48d4c06610d6f35ed8fece6249/lib/dkim/dkim-verifier.js#L228-L230.

This needs to be changed as follows:

-                            if (status === 'fail') {
+                            if (status.result === 'fail') {
                                status.comment = 'bad signature';
                            }

[titanism]

Attached is a redacted raw response from Gmail showing that it has passing DKIM, DMARC, and SPF for these exact same auto-response messages (which mailauth is incorrectly classifying as negative fail passes). test-gmail.txt

[titanism]

Another asynchronous improvement could be that usage of crypto.verify uses callback (or has a promise wrapper on it) per https://nodejs.org/api/crypto.html#cryptoverifyalgorithm-data-key-signature-callback as it would then use libuv's threadpool

[titanism]

I've attempted to debug and can't figure out why Gmail passes this DKIM signature and with crypto.verify we get a false bad signature. So that is one issue.

However I do know why SPF does not pass. It is because ptr is not implemented at

https://github.com/postalsys/mailauth/blob/9197684ded2ff44a2d5f2505f71ed8789699b93f/lib/spf/spf-verify.js#L393-L395

Can we implement this @andris9? And might you know why Gmail passes but mailauth doesn't?

[andris9]

Is the test.txt email tampered somehow? The To: header placement seems weird compared to other autoreply emails from Yahoo. When I run my own tests, I have no issues validating Yahoo autoreplies (see below). When I try to validate the test.txt email, then it fails, but it seems to be that the message is modified, so there is no way for me to check what is going wrong.

SPF check fails for a somewhat funny reason, Yahoo uses ptr: rules in the SPF record, mailauth skips these checks because of rfc7208 5.5:

5.5  "ptr" (do not use)

   This mechanism tests whether the DNS reverse-mapping for <ip> exists
   and correctly points to a domain name within a particular domain.
   This mechanism SHOULD NOT be published.

DKIM result for my own tests:

    {
        "signingDomain": "yahoo.com",
        "selector": "s2048",
        "signature": "XiQGeKaIeogTbnyO4lyab2QjWcjb4eErD5w+O75irQBP0rCnKpbcp67SFs6ByQoMDB+SFYvz499IAqARzshuE7nYFG73vM3WOiXSSO6W/GKCJKKx974ODqhqt7FGykOsNbRdZQbhNInM9s3pnxaNNbbsHVaqDIi9vXxqe420YHWN+hoZOJNn+16doHeztYZimhbyG9SQBvvtAXAHoZIiI+oSJcPo5ricjbEYuHeXCbUa9QHZfWOzUO2rM2i1riM3YM1MCFTuBl7m7Eo0cTLD3HCvLulUw1YAGszdeH1QYU/Phpyrrx7FV0Y9iR79h9Lu8kO0fs+9+DyANuCuHD0KcA==",
        "algo": "rsa-sha256",
        "format": "relaxed/relaxed",
        "bodyHash": "ssBQ2auA0tzBu2KCGffPnLXNzSawuhUM66bG+ouo0R8=",
        "bodyHashExpecting": "ssBQ2auA0tzBu2KCGffPnLXNzSawuhUM66bG+ouo0R8=",
        "signingHeaders": {
          "keys": "Date: From: To: Subject",
          "headers": [
            "Date: Fri, 29 Jul 2022 20:48:44 +0000 (UTC)",
            "From: Andris Reinman <andris.reinman@yahoo.com>",
            "To: andris.reinman@gmail.com",
            "Subject: Auto Response: Proovikiri"
          ]
        },
        "status": {
          "result": "pass",
          "header": {
            "i": "@yahoo.com",
            "s": "s2048",
            "a": "rsa-sha256",
            "b": "XiQGeKaI"
          },
          "aligned": "yahoo.com"
        },

[titanism]

It hasn't been tampered with, but I did notice the "To" placement being at the top as well, which is super weird.

[titanism]

If you want to test this out on your side, set up a free vanity email at https://forwardemail.net/my-account/domains/aliases/new?domain=hideaddress.net, e.g. andris@hideaddress.net. Then you can have the recipient be a Yahoo email address, which has an auto-responder turned on. This is what we did for forwardemailtest@yahoo.com.

[titanism]

Actually, we accidentally rewrote the "To" header for SRS during our tests. This is being fixed on our side.

I'll look into an alternative for the SPF check.

[titanism]

Do you think we can add PTR support @andris9 albeit it not being recommended for use?

[andris9]

Yes, I could add PTR support, I never imagined any sane organisation would actually use it, there’s a reason why it is discouraged

[titanism]

@andris9 We notified Yahoo's security team at security@yahooinc.com about this, but if you could add support in the interim that would be incredibly helpful (since others out there probably have DMARC with p=reject, no outbound DKIM, and thus failing SPF due to PTR usage). No rush either! We've fixed the invalid DKIM signature issue on our side already. 🙏

[titanism]

Updated title to reflect feature request. Thanks as always @andris9 👏 🚀

[andris9]

This is now fixed as of v4.0.0. Had to do a major bump due to some backward incompatible changes on counting SPF DNS requests.

$ mailauth --verbose --client-ip 66.163.189.147 --sender forwardemailtest@yahoo.com --helo sonic314-21.consmr.mail.ne1.yahoo.com --mta mx1.forwardemail.net test.txt
Reading email message from test.txt
....,
"spf": {
  "domain": "yahoo.com",
  "client-ip": "66.163.189.147",
  "helo": "sonic314-21.consmr.mail.ne1.yahoo.com",
  "envelope-from": "forwardemailtest@yahoo.com",
  "rr": "v=spf1 redirect=_spf.mail.yahoo.com",
  "status": {
    "result": "pass",
    "comment": "mx1.forwardemail.net: domain of forwardemailtest@yahoo.com designates 66.163.189.147 as permitted sender",
    "smtp": {
      "mailfrom": "forwardemailtest@yahoo.com",
      "helo": "sonic314-21.consmr.mail.ne1.yahoo.com"
    }
  },
 ...