dkim_verify identity/domain parsing edge case

celesteking commented 3 years ago

Something's up with dkim_verify plugin.

The sig, exactly as folded:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; s=key1; d=jameco.com;
 h=MIME-Version:Subject:From:Date:To:Reply-To:Content-Type:
 Content-Transfer-Encoding:Message-ID; i=TeamJameco@Jameco.com;
 bh=FIvqqGUc48Q8sDQdjPLbAVQWR5EAej/PksUGsIatiKE=;
 b=HGtJH4zaABqG4UsguLVXLSOeOS6RDSCsoV0fGtNjjYlxUDtWvwRVyfJVmCKt/GQyandJj4pkalP/
   /8fGv/VYeJbEQ6fI6oe0DVqY6tchgqrVSmb3isB3f7N14Vdpi2rAzk0T8j/TERu3Eqer2/tcqDe2
   LvxQ8Dsq7ZvX2OifLFg=

From: "Jameco Electronics" <TeamJameco@Jameco.com>

The log:

[DEBUG] [-] [dkim_verify] Found 2 DKIM signatures
[DEBUG] [-] [dkim_verify] {"identity":"unknown","selector":"key1","domain":"jameco.com","result":"invalid"}
..the other one verifies fine..

[dkim_verify] identity="unknown" domain="jameco.com" selector="key1" result=invalid  (domain mismatch)
[dkim_verify] jameco.com (domain mismatch)
[dkim_verify] identity="@esp1.co" domain="esp1.co" selector="app" result=pass 
[dkim_verify] [{"identity":"unknown","domain":"jameco.com","selector":"key1","result":"invalid","error":"domain mismatch"},{"identity":"@esp1.co","domain":"esp1.co","selector":"app","result":"pass","error":null}]

Can't get what's wrong. SA verifies it just fine.

System Info:

node v12, Haraka almost latest

celesteking commented 3 years ago

I have patched this, but now another thing poped up - lookup code can't deal with CNAMEd TXT records.

Mar  9 16:05:43 in7 haraka[6098]: [DEBUG] [-] [dkim_verify] info@periodicocubano.com: DNS lookup k1._domainkey.periodicocubano.com (timeout= 29s)
Mar  9 16:05:43 in7 haraka[6098]: [DEBUG] [-] [dkim_verify] info@periodicocubano.com: ignoring TXT record: dkim.mcsv.net

msimerson commented 3 years ago

https://www.npmjs.com/package/mailauth

celesteking commented 3 years ago

well for fucks sake. I've spent 1 month coding in milter support for opendmarc integration only for you to come and ruin everything. fuck fuck fuck.

celesteking commented 3 years ago

On second thought, it seems dns.resolveTxt is doing the right thing of following CNAME record and actually fetching the end result. It also looks like k1._domainkey.periodicocubano.com had recently changed their records to wrong values, that's why they were having issues. Crisis averted.

celesteking commented 3 years ago

Got another case of failed body hash computation -- not sure what's going on. Gmail signed the message. HK's dkim - fails to compute. SA verifies fine. The encoding is "foreign". See attached. test via ./bin/dkimverify --debug < dkf.eml , spamassassin -t -D dkim < dkf.eml . dkf.eml.gz

celesteking commented 3 years ago

The offender seems to be:

l = this.line_buffer.pop(line).toString('utf-8');
l = l.replace(/[\t ]+(\r?\n)$/,"$1");
l = l.replace(/[\t ]+/g,' ');

where non-utf8 gets replaced with notorious EFBFBD. Thus I'm wondering what's the easy way out of this foreign encoding drama.

msimerson commented 3 years ago

Issue #2182 could be related

celesteking commented 3 years ago

I've fixed the https://github.com/haraka/Haraka/issues/2929#issuecomment-797048285 , but still getting messages that even SA & python-dkim fail to verify. Like attached. shoplet-dkim-fail.eml.gz And even if HK puts wrong CRLFs, the dkim code will fix those. WTF!

celesteking commented 3 years ago

I just don't get how outlook managed to confirm this mail's body hash if none of avail tools to me are able to do it. That's directly from outlook, Haraka wasn't involved. midwest-out2.eml.gz

gene-hightower commented 3 years ago

Outlook re-writes mail, see the "Hotmail" comment here https://digilicious.com/mail.html

celesteking commented 3 years ago

Damn it. You were exactly right, outlook is mangling emails! What the actual fuck?! It looks like it's trying to "groom" html to be compatible with legacy clients (html entities like &zwnj not being understood, etc).

--- mid-original.eml
+++ mid-outlook.eml
@@ -78,12 +78,10 @@
 Content-Type: text/html; charset=us-ascii
 Content-Transfer-Encoding: 7bit

-<!DOCTYPE html>
-<html lang="en" xml:lang="en">
-<head>
-<title>Template</title>
+<!DOCTYPE html><html lang="en" xml:lang="en"><head>
+<meta http-equiv="Content-Type" content="text/html; charset=us-ascii"><title>Template</title>
 <!--[if (gte mso 9)|(IE)]><html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office"/><![endif]-->
-<meta charset="utf-8">
+
 <meta http-equiv="X-UA-Compatible" content="IE=edge">
 <meta name="x-apple-disable-message-reformatting">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
@@ -225,8 +223,8 @@
 </head>
 <body style="zoom: 100%;">
 <div style="display:none;font-size:1px;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;mso-hide:all;">Get started with your exclusive offer &#8211; a FREE GIFT! </div>
-<div style="display:none;font-size:1px;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;mso-hide:all;">&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;
-&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;&nbsp;&zwnj;</div>
+<div style="display:none;font-size:1px;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;mso-hide:all;">&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;
+&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;&nbsp;&#8204;</div>
 <table width="100%" bgcolor="#ffffff" cellpadding="0" cellspacing="0" border="0" role="presentation">
 <tbody>
 <tr>

celesteking commented 3 years ago

Just caught another mail, from walmart, and it's even more interesting. There's a dupe dot in it. It doesn't stand out of walmart crowd in any other way.

@@ -1121,3 +1110,3 @@
 t:15px!important}.productStrip .space{display:none!important}.productStrip =
-.extraPadding{display:none!important}.productStrip .bottomPadding{height:7p=
+..extraPadding{display:none!important}.productStrip .bottomPadding{height:7p=
 x!important}}.stars{width:28%!important}</style>

Removing the extra dot allows it to pass dkim. I refuse to believe there was a stray radiation ray hitting the sending MTA. The exact same host's been sending valid emails 2 days ago.

gene-hightower commented 3 years ago

There's a dupe dot in it. It doesn't stand out of walmart crowd in any other way.

This is called "dot stuffing." It is part of the SMTP DATA mechanism, code for this is down in the Haraka transaction.

celesteking commented 3 years ago

There's a dupe dot in it. It doesn't stand out of walmart crowd in any other way. This is called "dot stuffing." It is part of the SMTP DATA mechanism, code for this is down in the Haraka transaction.

Correct. And the message_stream has that double-dot already removed. Otherwise all the walmart messages would've failed.

celesteking commented 3 years ago

Arbitrary length keys should be supported.

[dkim_verify] invites@mailer.surveygizmo.com: got DNS record: v=DKIM1;k=rsa;p=MIIBIzANBgkqhkiG9w0BAQEFAAOCARAAMIIBCwKCAQIAxADjuRFdIA+CSlEQ09vU4S6uSZguPMK2Vhj8naLgyKYWVuA6zetoo6HBV0UhyQHhYoty
H57uybT/Po0VfHGS+4ecJr2Hxx5vayUXyJcZWk0FrADU701VJyWy2jVBxy83seL7i1E7z6p3cyvqj1XUja19ZEXzkRAiDYwkQQYbFSLU2DvYRUF6JLh7XnrP5rIErs7UpeVa03OPu66VtDoNdKKZ7S0QlLiU7+1t/Xpu2Le7qFIcl1rEbpqkmDeAKn2Xup5s0DSSYFUUg9WAuynIrWThoWvNTnvaZcuJXQ48veBG5dd7
lTOxLcgmg4Ah3fQvJBBa6SqGBvWhiMzH7H50bkkCAwEAAQ==
[dkim_verify] invites@mailer.surveygizmo.com: verification error: error:0906D064:PEM routines:PEM_read_bio:bad base64 decode

celesteking commented 3 years ago

Current issues

I have just confirmed (via wire capture) there's a Fedex host (mta.message.fedex.com) that's deliberately sending mails that fail DKIM bh verification.

Working fine

Update: just rechecked, and "dot stuffing" seems to be working just fine -- HK is removing the dot as per RFC. message_stream comes without any double-dots.

haraka / Haraka

dkim_verify identity/domain parsing edge case #2929

Current issues

Working fine