search

trusteddomainproject / OpenDMARC

This is the Trusted Domain Project's impementation of the DMARC protocol libary and mail filter, called OpenDMARC. A "milter" connects to unix-based mailers (originally, sendmail, but now many) and provides a standard filtering API.
Other
102 stars 55 forks source link

dmarc=fail with develop branch but works with testing #144

Closed gene-git closed 3 years ago

gene-git commented 3 years ago

Split this out of issue #95 as suggested by @thegushi

Same test mail from gmail with testing vs develop branches:

For good measure, re-installed testing version and all works. Repeated the test 2nd time to confirm with same outcomes

no other changes other than the build of opendmarc.

Each build was packaged into Arch package and installed using pacman - the difference simply being the git branch. In both test cases the envelope from matches the from header and matches dkim d=gmail.com and both dkim and spf pass.

Test server running Archlinux fully updated (using testing repos)

martinbogo commented 3 years ago

Gene,

Here's a further mystery for you. I repeated your setup, but instead of Arch linux I used Debian 10 as my base.

... and it works fine, both in the develop and testing branches.. So, we've got to dig deeper. @thegushi is trying to put together an Arch linux VM to reproduce your results.

On Wed, Mar 24, 2021 at 8:55 AM Gene @.***> wrote:

Split this out of issue 95 [1] as suggested by @thegushi https://github.com/thegushi

Same test mail from gmail with testing vs develop branches:

For good measure, re-installed testing version and all works. Repeated the test 2nd time to confirm with same outcomes

no other changes other than the build of opendmarc.

Each build was packaged into Arch package and installed using pacman - the difference simply being the git branch. In both test cases the envelope from matches the from header and matches dkim d=gmail.com and both dkim and spf pass.

Test server running Archlinux fully updated (using testing repos)

  • gcc 10.2.0-6
  • postfix 3.5.9-2
  • binutils 2.36.1-2
  • kernel 5.11.8-1-stable

[1] https://github.com/trusteddomainproject/OpenDMARC/issues/95`

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/trusteddomainproject/OpenDMARC/issues/144, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB5KKNE3LATVD3GY5R2GQDTFHVNNANCNFSM4ZXKAZXQ .

gene-git commented 3 years ago

Thanks for doing that - and well darn and shucks that's no fun at all !!!

Additional info, I'm using opendkim / openarc milters.

But now i'm even more stumped. And, since i repeated the test with identical results - and fail was immediate - it doesn't smell to me like one of those annoying subtle memory bugs ( that we're supposed to eliminate moving to rust).

Very perplexed.

thegushi commented 3 years ago

Can you send along your config.log? (Again, if you’d rather not have such things in public view, you can email it to me)

-Dan

On Mar 24, 2021, at 5:37 PM, Martin Bogomolni @.***> wrote:

Gene,

Here's a further mystery for you. I repeated your setup, but instead of Arch linux I used Debian 10 as my base.

... and it works fine, both in the develop and testing branches.. So, we've got to dig deeper. @thegushi is trying to put together an Arch linux VM to reproduce your results.

On Wed, Mar 24, 2021 at 8:55 AM Gene @.***> wrote:

Split this out of issue 95 [1] as suggested by @thegushi <https://github.com/thegushi https://github.com/thegushi>

Same test mail from gmail with testing vs develop branches:

For good measure, re-installed testing version and all works. Repeated the test 2nd time to confirm with same outcomes

no other changes other than the build of opendmarc.

Each build was packaged into Arch package and installed using pacman - the difference simply being the git branch. In both test cases the envelope from matches the from header and matches dkim d=gmail.com and both dkim and spf pass.

Test server running Archlinux fully updated (using testing repos)

  • gcc 10.2.0-6
  • postfix 3.5.9-2
  • binutils 2.36.1-2
  • kernel 5.11.8-1-stable

[1] https://github.com/trusteddomainproject/OpenDMARC/issues/95`

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/trusteddomainproject/OpenDMARC/issues/144 https://github.com/trusteddomainproject/OpenDMARC/issues/144>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5KKNE3LATVD3GY5R2GQDTFHVNNANCNFSM4ZXKAZXQ https://github.com/notifications/unsubscribe-auth/AAB5KKNE3LATVD3GY5R2GQDTFHVNNANCNFSM4ZXKAZXQ> .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/trusteddomainproject/OpenDMARC/issues/144#issuecomment-806273539, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIWKKCAQOGIXVKLYUE4JXTTFKAVHANCNFSM4ZXKAZXQ.

gene-git commented 3 years ago

Sure will do Dan - need to rebuild with develop. Thinking out loud - I don't think I do a 'make clean' - i just do config and make - I'll check the package build to confirm.

Sometimes makefiles don't catch every dependency which can lead to odd results. I'm thinking i should make sure build is really clean as well. Then retest.

Anyway - will get back to with config log - do you want config.log from both testing and develop? Oh - machines are now all on kernel 5.11.9 stable as of today - not key I know, just FYI

thegushi commented 3 years ago

Honestly, just diff them and see if you get any smoking guns.

-Dan

On Mar 24, 2021, at 6:08 PM, Gene @.***> wrote:

Sure will do Dan - need to rebuild with develop. Thinking out loud - I don't think I do a 'make clean' - i just do config and make - I'll check the package build to confirm.

Sometimes makefiles don't catch every dependency which can lead to odd results. I'm thinking i should make sure build is really clean as well. Then retest.

Anyway - will get back to with config log - do you want config.log from both testing and develop? Oh - machines are now all on kernel 5.11.9 stable as of today - not key I know, just FYI

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/trusteddomainproject/OpenDMARC/issues/144#issuecomment-806284184, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIWKKEZQN5GZTEUVG3JLY3TFKEITANCNFSM4ZXKAZXQ.

gene-git commented 3 years ago

sure no prob

thegushi commented 3 years ago

So I’m running a git diff between develop and testing, trying to play the mental game of “where could it be"

Here’s the main things we focused on in develop (it’s a point release, after all):

0) Documentation cleanup, including the changeover to GitHub.

1) The CVE responses that led to a new check_domain function (effectively, discarding any invalid d= domains). It’s possible there’s some string function that’s linux-specific that fails for you. There’s one other XML vuln that we fixed, but the thing that resulted in the majority of the code was the above.

2) There’s a potential off-by-one thing in a use of strcpy (or is it strlcpy) that we wanted to look at again. You can find Bicknell’s bug on this in the tracker. (Leo and I are former coworkers, I know him well).

3) Getting the HoldQuarantinedMessages function in there. Because the milter was filling people’s hold queues and we don’t want to violate principle of least astonishment. You don’t have that feature turned on.

4) More tweaks to the check_multi functions, which shouldn’t hit you with gmail since there’s only one from.

I’ve got an Arch Linux vm on my laptop (using a local-only network), but I’m going to start the upload to my colo this evening.

Arch is…remarkably unfriendly to someone who’s never used it before, btw. Any installer that says “here’s fdisk, have fun”...

-Dan

On Mar 24, 2021, at 6:00 PM, Gene @.***> wrote:

Thanks for doing that - and well darn and shucks that's no fun at all !!!

Additional info, I'm using opendkim / openarc milters.

But now i'm even more stumped. And, since i repeated the test with identical results - and fail was immediate - it doesn't smell to me like one of those annoying subtle memory bugs ( that we're supposed to eliminate moving to rust).

Very perplexed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/trusteddomainproject/OpenDMARC/issues/144#issuecomment-806281386, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIWKKHN5HKHQKS6H2VPB4DTFKDMFANCNFSM4ZXKAZXQ.

gene-git commented 3 years ago

yeh arch can be a bit terse lol. But i do like rolling release. I did a fresh build of both branches (added make clean to build script) and will rerun same test - sent you diff -y of configs.

gene-git commented 3 years ago

With clean from scratch build - develop branch working fine for same test. I think therfore i have to chalk this up to a bad build:

good news - as it made no sense as everything was dmarc aligned and yet seemingly failing ... and I was alone in seeing this.

bad news - also bad as I've wasted your time - and am very sorry about that ... I've changed the build script to always do a clean build from scratch now.

Again I apologize for not figuring this out before bothering you all.

thegushi commented 3 years ago

No worries, it was fun problem solving. One of the things we want in 1.5.0 is MOAR UNIT TESTS! and MOAR_FILTER_TESTS!

I’ll close this out when I log back into Github

On Mar 24, 2021, at 6:58 PM, Gene @.***> wrote:

With clean from scratch build - develop branch working fine for same test. I think therfore i have to chalk this up to a bad build:

good news - as it made no sense as everything was dmarc aligned and yet seemingly failing ... and I was alone in seeing this.

bad news - also bad as I've wasted your time - and am very sorry about that ... I've changed the build script to always do a clean build from scratch now.

Again I apologize for not figuring this out before bothering you all.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/trusteddomainproject/OpenDMARC/issues/144#issuecomment-806302983, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIWKKGH7WN6GK5U5EQWTGDTFKKCRANCNFSM4ZXKAZXQ.

gene-git commented 3 years ago

Thanks dan - i'll continue running builds and testing with develop branch.

gene-git commented 3 years ago

oops - added comment and deleted by mistake -

My recent test was flawed - develop branch does still have dmarc=fail and testing has dmarc=pass - as first reported. When I did test a few minutes ago - i installed the develop package but forgot to restart the opendamrc daemon - so test simply re-ran with testing build instead of develop.

Once i actually started the develop daemon - it fails - this was a fully clean build.

So ... i re-opened the issue ... duh me.

gene-git commented 3 years ago

Update - user config error which Dan, impressively, figured out somehow - thank you so much,

Summary: On the server running opendmarc the AuthservID of the companion opendkim milter is different - after adding the dkim AuthservID to the list of TrustedAuthservIDs in the opendmarc config - all works correctly. Not quite obvious perhaps why testing was happy while develop was not when relying on SPF instead of dkim, - but fixing my config allowed the dkim=pass to be used and thus passing dmarc.

thegushi commented 3 years ago

Okay, I've managed to replicate this. It takes an insane setup which Gene managed to expose by accident.

Basically, if you have an AuthServID that differs from your TrustedAuthServID, then you're going to hit this failure mode.

Yes, specifically, this would seem to mean that it's possible to configure opendmarc to NOT TRUST ITSELF

Case in point, my opendmarc was configured on an Arch Linux system thusly:

SPFIgnoreResults true
SPFSelfValidate true
AuthservID archlinux.gushi.org
TrustedAuthservIDs foobar.com

Which means even though I have DKIM and the like in the stream, those would be overlooked by the filter.

Relevant mail headers:

DMARC-Filter: OpenDMARC Filter v1.4.0 prime.gushi.org 12RMvAqJ059028
Authentication-Results: archlinux.gushi.org; dmarc=fail (p=none dis=none) header.from=gmail.com
Authentication-Results: archlinux.gushi.org; spf=pass smtp.mailfrom=gushimailtest@gmail.com
DKIM-Filter: OpenDKIM Filter v2.10.3 prime.gushi.org 12RMvAqJ059028
Authentication-Results: prime.gushi.org;
    dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b=oWZVIwBY

Here's the thing about this that's broken. If we do not trust our own header, we shouldn't get dmarc=fail

RFC7489 Section 11.2:

Meaning of "pass": No DMARC policy record was published for the aligned identifier, or no aligned identifier could be extracted."

(The second is true in this case. Because of the mismatched AuthServID's, we have extracted the identifier but we do not believe it. Thus, we basically have no trusted headers to base a judgement on, so we must pass it, just as if we had no SPF header at all).

I'm going to refer this to the rest of my team to look at in this case to see if it looks worth fixing, or if it's enough of a corner case to ignore for now.

thegushi commented 3 years ago

And yes, against the testing branch, we get dmarc=pass

DMARC-Filter: OpenDMARC Filter v1.4.0 prime.gushi.org 12RNNFZt061729
Authentication-Results: archlinux.gushi.org; dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: archlinux.gushi.org; spf=pass smtp.mailfrom=gushimailtest@gmail.com
DKIM-Filter: OpenDKIM Filter v2.10.3 prime.gushi.org 12RNNFZt061729
Authentication-Results: prime.gushi.org;
    dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b=Ml0xHGw9

We have our regression, and I suspect Arch Linux was just a red herring. We would have this on any platform.

gene-git commented 3 years ago

Thank you Dan for the detailed analysis and glad you got it replicated even!!

mskucherawy commented 3 years ago

We found the problem here. Thank you very much for reporting it.

A function in libopendmarc receives a parameter called "domain". On the "develop" branch, we added a check for a syntactically valid domain name here to be sure that a malformed domain can't be sent into the filter to confound it into doing the wrong thing. However, the thing passed into that function is actually the entire envelope sender, so naturally every "domain" it receives is rejected because no valid domain contains an "@" sign. Moving the check down a few lines past where the domain name is extracted, and testing that instead, fixed the problem.

A commit fixing this will appear shortly.

gene-git commented 3 years ago

That's excellent - really glad you tracked it down and sorted it out.

Thank you!!