search

domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
1.01k stars 219 forks source link

TLS Reports from microsoft.com are not parsed #561

Open makuartur opened 1 month ago

makuartur commented 1 month ago

The TLS report from microsoft.com wasn't parsed and was moved to the invalid folder From: tlsrpt-noreply@microsoft.com

Subject: _Report Domain:
Submitter: microsoft.com
Report-ID: ^[0-9]{18}+_

Body: _This is an aggregate TLS report from microsoft.com

Microsoft respects your privacy. Please review our online Privacy Statement.

Microsoft Corporation
One Microsoft Way
Redmond, WA, USA 98052_

Attached filename:
microsoft.com!domain_name!1725840000!1725926399!report_id.json.gz

Content of the file: 

{
  "organization-name": "Microsoft Corporation",
  "date-range": {
    "start-datetime": "2024-09-09T00:00:00Z",
    "end-datetime": "2024-09-09T23:59:59Z"
  },
  "contact-info": "tlsrpt-noreply@microsoft.com",
  "report-id": "report_id+domain_name",
  "policies": [
    {
      "policy": {
        "policy-type": "sts",
        "policy-string": [
          "version: STSv1",
          "mode: enforce",
          "mx: aspmx.l.google.com",
          "mx: alt1.aspmx.l.google.com",
          "mx: alt2.aspmx.l.google.com",
          "mx: alt3.aspmx.l.google.com",
          "mx: alt4.aspmx.l.google.com",
          "max_age: 604800"
        ],
        "policy-domain": "domain_name"
      },
      "summary": {
        "total-successful-session-count": 2,
        "total-failure-session-count": 0
      }
    }
  ]
}

parsedmarc version: 8.14.1

seanthegeek commented 1 month ago

Running the raw json file you provided through parse_report_file() parses the report correctly. So, the issue must be with the email itself. I don't have any email samples from Microsoft to test with. Can you provide one?

makuartur commented 1 month ago

Will this option suit you? Delivered-To: admin@fake.domain.name Received: by 2002:a05:612c:1a52:b0:48e:c7f6:a1f with SMTP id hu18csp1070346vqb; Thu, 12 Sep 2024 13:42:36 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH6Cy3MXLWC6ihmN3fU8IWcv+413i8hf9sdXzU8zeg2i0udUB4CdrelYSfEipFvr531u0Qy X-Received: by 2002:a05:6a21:1190:b0:1cf:6baf:61c0 with SMTP id adf61e73a8af0-1cf764c29e6mr5901032637.44.1726173756013; Thu, 12 Sep 2024 13:42:36 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1726173756; cv=pass; d=google.com; s=arc-20240605; b=PdN1ae9sPwdb2g3VeZSUMKTpwv9yEvArk2scl+IFUs8RWqKJJ1myqmvaIMSdFQnvCY UzP8xLIwYbdjhNTLGbWy+QIqnCpAd9+vrwPPnAhpvRKKtage4yM7ZHBSi/kbld4KRJNv LagNakHURiLIec92qPs7EJqbtSvtqE33sVOo5+26RVSge8eJW3z/gtdFrg4HOx6qM9kv hQ/EV/L9O/WdL+M6/q0ZuxonyNe2JSYr7wrj5U3iAL0eUTSV6VW/9tXbmbgil6zUigSX 5/d2ShvKqU5oGJzzBOdxbHMoOkVgHjgT3wQ3/mRjjekgHmr7FV4R7FcfGGBWLrnIRapv cRxQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=message-id:mime-version:tls-report-submitter:tls-report-domain :subject:date:to:from:dkim-signature; bh=hMoLhrNBXNEU8Zpp9QAId7yWfsZ80FBUxqB9dt0ycRg=; fh=NlfXymrgzSwYC67628bBfGK4hKLRfJcdy8QuPYRp5f4=; b=Iq68YMfs5wcf9cF18VW2zdHpWuY4Eiyx+fqSD+iDGJd1uPo4yZh1bk5IKd7+WYSIzK v+D3cU9mC1TmF9q5HfIzY5tepDUNUUv1RRkk7J2H7htYowyrvCYIwN84xZQEZKAsrpMc a2UQcjraXYbdY8XiHtNLhVODhJCcQsCqlCgl98HhLIdyaXgbPnsv1scps0Vz1uAi5351 Qa+G00IpehzRf12+ZxRrXqtlsY95Yc8gSaEHyX9W9swx+5CnM4p6HS9TcL+75iYVjY0Y GDnUDcwu2mvj0lL7iWVqFYgPCX8uuDArqqMuyqOe47PUrTVQsHXsnlZHKFCkvrFKwsFZ 1mbg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@microsoft.com header.s=selector2 header.b=MnrBBYSL; arc=pass (i=1); spf=pass (google.com: domain of tlsrpt-noreply@microsoft.com designates 2a01:111:f403:c000::1 as permitted sender) smtp.mailfrom=tlsrpt-noreply@microsoft.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=microsoft.com Return-Path: tlsrpt-noreply@microsoft.com Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazlp170100001.outbound.protection.outlook.com. [2a01:111:f403:c000::1]) by mx.google.com with ESMTPS id 41be03b00d2f7-7db1fe308b6si3314850a12.780.2024.09.12.13.42.35 for admin@fake.domain.name (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Sep 2024 13:42:35 -0700 (PDT) Received-SPF: pass (google.com: domain of tlsrpt-noreply@microsoft.com designates 2a01:111:f403:c000::1 as permitted sender) client-ip=2a01:111:f403:c000::1; Authentication-Results: mx.google.com; dkim=pass header.i=@microsoft.com header.s=selector2 header.b=MnrBBYSL; arc=pass (i=1); spf=pass (google.com: domain of tlsrpt-noreply@microsoft.com designates 2a01:111:f403:c000::1 as permitted sender) smtp.mailfrom=tlsrpt-noreply@microsoft.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=microsoft.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KNelPmz4Bk31LbzUSMsvaJnGO7k4W9t9E3B0qgwwab0Rs59A5s6lMokpa1WMnFZvjehmT9Ienm8kIG0z+wkLwWU2nGBBYPTcvtH7LelOkNPV2DYgw3F+J2meR3KqQ0yoy2zCKJjhCna1KI2sQd+7CeFZXNXMrjRGwVLmdPoZeuTSCKxzF8A1MOOjoGv3tc1KXTSvAAbKyxX2pXhhy+PkkJI7F6433+9gxGCFm4r9FISUR8ov+ky9/aoioG4Ju7vt3tBRs/Kw28j7FzYXUvaHOtLc6L1aXwl/2Ylg4Hwz19TpcwnfWs+tKHitEJgVsAO2m1c11Lyq63rqRpbkM6Hdzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hMoLhrNBXNEU8Zpp9QAId7yWfsZ80FBUxqB9dt0ycRg=; b=xOAPZWjLxfM357qsInlVxF1tGQXUyJuEFKDYrG3LbHByjl81tN/GGn3lWbR0VJldJLsrkf4thku11X8vbt35tdAJMHibvXx+85FnYGPiPqOe8PSIcGCfhJYHhLBWW+ZjZTw2Q2ptwo2/IpVstVemeZ2tp1gYZh/XmtI4v3JBhl325q/Nd3Gd5u2trmCRyFKSolXO4oGvEh/fC1KA1VNO9WIXnOgd6JK2twX9qPsyECJvEE3uRwMAz9nAdZyS7sXT526vEevYu8HoHvptejFVSgo3QRuq4UihHPvWuEOa/sQWOJ1YUDxg67t46nCkKsB5mEPrHZWMEHynCfCRpHjfzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none action=none header.from=microsoft.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hMoLhrNBXNEU8Zpp9QAId7yWfsZ80FBUxqB9dt0ycRg=; b=MnrBBYSL8p8lsu+H6coxDIm2W8mcubWuY095SpN/1hcPWMrQ0XfHUEfIxhxV2XZqCC2Dbmsmv9CVBlGcijye5Y0RboUfu6YeFPZr2X+et4gRKpZP4akxVgemXo9SOf22dmnePkIY6pPWDOyJxp8LEh8ySE7EJbsPNsmNlSQsDa4= Received: from MN2PR00CA0008.namprd00.prod.outlook.com (2603:10b6:208:224::21) by IA2PR21MB4323.namprd21.prod.outlook.com (2603:10b6:208:4b0::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.6; Thu, 12 Sep 2024 20:42:32 +0000 Received: from BL2NAM06FT015.Eop-nam06.prod.protection.outlook.com (2603:10b6:208:224:cafe::af) by MN2PR00CA0008.outlook.office365.com (2603:10b6:208:224::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7999.0 via Frontend Transport; Thu, 12 Sep 2024 20:42:31 +0000 X-MS-Exchange-Authentication-Results: spf=none (sender IP is 172.179.146.41) smtp.mailfrom=microsoft.com; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=microsoft.com; Received: from 104.47.53.36 (172.179.146.41) by BL2NAM06FT015.mail.protection.outlook.com (10.152.107.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.4 via Frontend Transport; Thu, 12 Sep 2024 20:42:31 +0000 From: tlsrpt-noreply@microsoft.com To: "admin@fake.domain.name" admin@fake.domain.name Date: Thu, 12 Sep 2024 20:42:31 +0000 Subject: Report Domain: fake.domain.name Submitter: microsoft.com Report-ID: 111111111111111111+fake.domain.name TLS-Report-Domain: fake.domain.name TLS-Report-Submitter: microsoft.com MIME-Version: 1.0 Message-ID: 111111111111111111+fake.domain.name@fake.domain.name Content-Type: multipart/report; boundary="a737a3b9-5ea5-40e8-a971-a67ecbb2993e"; report-type=tlsrpt Return-Path: tlsrpt-noreply@microsoft.com X-MS-TrafficTypeDiagnostic: BL2NAM06FT015:EE_FirstParty-TlsRpt-V3-System|IA2PR21MB4323:EE_FirstParty-TlsRpt-V3-System X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ae339a39-51d3-4824-723f-08dcd36b6cb3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|61400799027; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?X2r3WPqkzvTm/A5lM6ZYm9IR4lza9Y1O8dF0fsIkNYvqmwis6K/K9Nb5JQm0?= =?us-ascii?Q?MNbSgbbreHa75V1ewsp3C+faU6o6HW+mi6Qk+brxHpHoNYP96fQvyRlG3liQ?= =?us-ascii?Q?s+2xvt6S8ud9rDGXKHA7UQ/YfVQShsci7iKmgQ729XhBqq3YzaM3VkV/mdNF?= =?us-ascii?Q?d2o/Pnl/C63Plyc118ZCxKRT8ezmnXY2UwHrDN5N8kwAIO63Tc+4TYDRiNKJ?= =?us-ascii?Q?h9M4iPAiUI9nEXXyoZg0xlcYAVL7iiT2sJO534MLsxcD3Z/UudaVjhjBcs4u?= =?us-ascii?Q?59FpMsXxJxDLQs9OMvwaH2smwQE3VFf9QL3HNc+iT9G/frViGaEZen2rNu0E?= =?us-ascii?Q?X9uHQbqIj+X2Nk3TX1ytsRLfrueUPTgFSKEqEuF8qZErfzbd5qtCA/2CUHN2?= =?us-ascii?Q?GlwLjoKErzWNOzkxwSS1dy/FquA4cCSpSvJWiJjSmIZJ4kfJqcNiwZb4AfZt?= =?us-ascii?Q?g63Sol3MBTC6DPlr/PuXWHgktCwMsy0ElbGmFmc0I0f00ts7h2Jdjc9VCP2c?= =?us-ascii?Q?+Qz/FazYXoeNqg/KC72u3I5uJkZtrmnFzjde3uD4MHYUdMnAgfhbbZZpmUPw?= =?us-ascii?Q?QNVUZX3GHuJZcri4QkAANPaVhrbidsQflLpPTVBJE7BflmKCDzsUib6SW0Xi?= =?us-ascii?Q?lk71c9/iATSlm/LdXpRuxDqzAndEoIm7dllqOSWDbqZQXP7f/eEnW0zBxWqB?= =?us-ascii?Q?i1a+9sphIyiPSxcaMLcu4+aH9MpU4sX9FWsOJYoZBlF8mhUjC455pTzD55Dp?= =?us-ascii?Q?uihDJ27e5is+BApIKnr0aXFpGH7aFD6IHPKUEV5BkK6PXfPgb/GMQhpHuG66?= =?us-ascii?Q?fQ3c3nTOgNqpTbxr04FKFIXhRBDS3S8d5y4w9+cvrm6r0yY3M8rjn5DEVDH+?= =?us-ascii?Q?AYiCH9vyTJL3Rsx0TaHrtoY37Aohv1etjpcVTt3WhzFnS4l6T8NpIr1X1Y4z?= =?us-ascii?Q?bfTqNqzAuQi76RGRyGv4TBVAY90qZ7pzKAEBfI7HSxq5bqCP9k9lPNM7lfMF?= =?us-ascii?Q?GvS/bDKXvgD2C9g5qvh/6MgQBmbXijP5ox5xePZTBoCnWYxZKhg/yBZ07vtp?= =?us-ascii?Q?5irGFQQhRb69DdGe3crb4XI5CRq8gIcm4aeAUViE3Uxe1aZivAZVS7b1NXZI?= =?us-ascii?Q?RltEq1HXq6vuuUfLfRijainR5bRXTgRmfmHYme6Xtl4qvmNOb2CHQ7hD3b8J?= =?us-ascii?Q?VA5k/aSrsqyoXbJRrGsy3W7BM6lvRk7jYPo+zjkSTgQtxaIFvxmXZlQMThqn?= =?us-ascii?Q?Plt0XK/8rE+eZbal5+j1xQd6LLA1eaKcwuKorGZQXSPtOTgSIKg/c/a9Ma9f?= =?us-ascii?Q?TTog5r3+7Yj+MKQgC3CICziD/39VReTWpbrlksjchrG0mgniN3z/6IYuCRY2?= =?us-ascii?Q?5UQKkEYYMoxfRl98qiCBSHc3nogv?= X-Forefront-Antispam-Report: CIP:172.179.146.41;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:104.47.53.36;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(61400799027);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: NY8NAfnSjnGmunwEE1ByyaAv8ChGS2wdJ5egQFwMw9HoQzI4aNX2u+X+MclJj4vFU3Bi+JrH2EkBcg+lWT+vg69gOPjdUPAe91TOLB4gENtuSUyTjijBbiKIqe9USw1vKrob1kmH3cwpusMsqQ8jZWVir9dVi5zvUFAGaGFSLlvvtdkBk8qc/h3IwJ1JaTWAbvymLIDfPyCM9ut/0t1UyCcHBDt/NptX7lW+llvP7CUOCOzudx7F4v6czUW1KYl2fEzAnFeUfYRo0gGJgEVKCD6GAS28nzRo8H8EyBK7zvZt5pyy5QHak1Wz8mhJIU0DJ45C/EVgi+SvXl1T9ZagbC2InOdA2nmCqNPNbz272aizHkd0YmiY+S4bhgnsa7aSad7KaKoD/hRI9c63MQR+IRhKV1Yukj0xC+aCM1g0CVPdtOYfQUwujNaVx8kANKCmKjLcsyCEhaA/5MUotYZepPqvW7t3zRTFE2W41O770oGqczpNjk0HtpAfBLBXeYBY X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Sep 2024 20:42:31.7070 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ae339a39-51d3-4824-723f-08dcd36b6cb3 X-MS-Exchange-CrossTenant-Id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=72f988bf-86f1-41af-91ab-2d7cd011db47;Ip=[172.179.146.41];Helo=[104.47.53.36] X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: TreatMessagesAsInternal-IA2PR21MB4323.namprd21.prod.outlook.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA2PR21MB4323

--a737a3b9-5ea5-40e8-a971-a67ecbb2993e Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

This is an aggregate TLS report from microsoft.com


Microsoft respects your privacy. Review our online Privacy Statement<= /em>

Microsoft Corporation
One Microsoft Way
Redmond, WA, USA 98052

--a737a3b9-5ea5-40e8-a971-a67ecbb2993e Content-Type: application/tlsrpt+gzip Content-Description: microsoft.com!fake.domain.name!1726012800!1726099199!111111111111111111.json.gz Content-Disposition: attachment; filename="microsoft.com!fake.domain.name!1726012800!1726099199!111111111111111111.json.gz" Content-Transfer-Encoding: base64

H4sIAAAAAAAEAH2STWvDMAyG/0rwdXVwPpa1Pg123qk9dZThOU4wxFawldKs5L9PSdsxBi0ILPt9 ZL8SPjMIrfL2W6EFz71yhkn2bnWACA0mbxB6CIvIVqxWaHhQviXozCKqgHw+Q7uU5SIvudjwLNsJ IZfYU5Xx9R0qL+TzhmLPphXT4FFp5NY3QBh2MfTIPQTTd+Oru1lKNTi6lE6BXrc1oVlRvIiqzMts vc43oqqKpzi6LwtoddpYonvorLYmMvlxvmzGuYNLxnHsZ2MR4w0decRgfUs8O5oQqX2ZbHfbY0aE g9rIxJDNoM28P8lExd6d0i5tAdrOXD0uQodZ+kjNH6rFQ7W8o6rTp2rJYyXKtRDs8NtVDU5ZP/f6 dz40/Dg4p8IyFARUHY+D1ibGZqCU1vlzaBg8MpmvrkijbDcE818X03SYfgCxXYk2WAIAAA==

--a737a3b9-5ea5-40e8-a971-a67ecbb2993e--

makuartur commented 1 month ago

@seanthegeek, is the answer provided appropriate?

seanthegeek commented 1 month ago

Yes. The problem is a mistake in the email headers when Microsoft is generating the tlsrpt email. You can verify this by pasting the content you provided into a .eml file and then opening in in an email reader like Thunderbird. The email content will be blank.

The mistake is here, where the Content-Type header is split over two lines without indenting the second line.

Content-Type: multipart/report;
boundary="a737a3b9-5ea5-40e8-a971-a67ecbb2993e"; report-type=tlsrpt

If you combine the content into one line

Content-Type: multipart/report; boundary="a737a3b9-5ea5-40e8-a971-a67ecbb2993e"; report-type=tlsrpt

Or indent the second line

Content-Type: multipart/report;
  boundary="a737a3b9-5ea5-40e8-a971-a67ecbb2993e"; report-type=tlsrpt

Then the email will be successfully parsed by Thunderbird and parsedmarc.

I'll see if I can find someone at Microsoft to address this.

seanthegeek commented 1 month ago

@makuartur Actually, looking at this again, it looks like GitHub removed all of the indents when you pasted in the sample as test, Please save the sample as a file, then drag and prop that file into the comment box to make it an attachment.

makuartur commented 1 month ago

mail.txt Of course, here it is in the file. @seanthegeek

seanthegeek commented 1 month ago

Hmm. the parsedmarc CLI parsed the email correctly, so I'm not sure why it would be moved to the invalid folder. Can you try updating to the latest release of parsedmarc, moving one of the emails from the invalid folder back to the inbox and see if the same thing happens again?