search

antedebaas / Viesti-Reports

DMARC & SMTP-TLS Reports processor and visualizer and BIMI file hoster
https://docs.viestireports.com/
GNU Affero General Public License v3.0
82 stars 16 forks source link

Update GetReportsFromMailboxCommand.php #111

Closed d--j closed 5 months ago

d--j commented 5 months ago

Catch \Throwable instead of \Exception to also catch PHP errors (like passing null to a string argument)

See https://www.php.net/manual/en/class.throwable.php

Catching \Exception will not catch error conditions where required email header or report fields are missing. Such errors get thrown as \Error.

antedebaas commented 5 months ago

I wonder what mailserver you are using that does not provide a mailid?

d--j commented 5 months ago

Hi @antedebaas,

All valid DMARC and SMTP-TLS reports will have a Message-ID header. But the automatic import process should not error out on other emails that are missing this header.

I wonder what mailserver you are using that does not provide a mailid?

I'm running a Postfix/Dovecot setup – Postfix is not configured specailly (uses the default) always_add_missing_headers = no and remote_header_rewrite_domain =

Anyway even when I would re-configure these options to add the missing headers (and then maybe break DKIM signing of emails without a Message-ID header that includes the Message-ID header) I think we should handle the edge case that there is no Message-ID in an email since the IMAP server may have other email sources other than Postfix (e.g. Sieve scripts, direct LMTP delivery).

antedebaas commented 5 months ago

fair.

But i'll be implementing it in a slightly different way as catching Throwable directly is bad practice.

d--j commented 5 months ago

But i'll be implementing it in a slightly different way as catching Throwable directly is bad practice.

Sure. You might want to sprinkle in some more null guards, too.

Here are two samples of in-the-wild DMARC/SMTP-TLS reports that could not be parsed with the current code:

Missing Message-ID:

Return-Path: <dmarc.report@interia.pl>
Delivered-To: reports@mailing-services.net
Received: from sogo.saasweb.net
    by sogo.saasweb.net with LMTP
    id cgGxAMLjV2Z/ZzQArVOHLA
    (envelope-from <dmarc.report@interia.pl>)
    for <reports@mailing-services.net>; Thu, 30 May 2024 04:26:10 +0200
Received: from smtp.firma.interia.pl (sm01.firma.interia.pl [217.74.65.192])
    (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
     key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384)
    (No client certificate requested)
    by sogo.saasweb.net (Postfix) with ESMTPS id F2BE740180
    for <reports@mailing-services.net>; Thu, 30 May 2024 04:26:08 +0200 (CEST)
Authentication-Results: sogo.saasweb.net;
    dkim=pass header.d=interia.pl header.s=biztos header.b=GSVd2B34;
    spf=pass (sogo.saasweb.net: domain of dmarc.report@interia.pl designates 217.74.65.192 as permitted sender) smtp.mailfrom=dmarc.report@interia.pl;
    dmarc=pass (policy=none) header.from=interia.pl
Received: from [10.42.19.229] (gw.paas.interia.pl [185.69.192.31])
    by smtp.firma.interia.pl (Postfix) with ESMTP id 55E5C2047E
    for <reports@mailing-services.net>; Thu, 30 May 2024 04:25:51 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=interia.pl; s=biztos;
    t=1717035951; bh=YFCLM3NGTr7gzmo4iIN0EEqqafhStZm43YrSN1rQgR8=;
    h=From:To:Subject:MIME-Version:Content-Type;
    b=GSVd2B34PmetldVZtVYVkDSf6vOYbOyRAzg8/OtTLkOE9bTcYUITpCtusneWpomlQ
     y2QqVprhjGCaixIUdncMQkBZz5xPRsFUgnqR9y3CaxKDoxNjvYw7Sd06lQ0/+HXXCR
     kwJ5oVsd2nkND83v1mThrX25h5U/EJJtHlOrMtgQ=
From: dmarc.report@interia.pl
To: reports@mailing-services.net
Subject: Report Domain: jagszent.de
 Submitter: interia.pl
 Report-ID: 125472898370097792358345580141597375534
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============4385374180968755410=="
en-us: 
X-Spam-Status: No, score=2.59
X-Spam-Level: **
X-Spamd-Bar: ++

This is a multipart message in MIME format.
--===============4385374180968755410==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

This is an aggregate report from interia.pl.
--===============4385374180968755410==
Content-Type: application/gzip
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="interia.pl!jagszent.de!1716940800!1717027200.xml.gz"

H4sICK/jV2YC/2ludGVyaWEucGwhamFnc3plbnQuZGUhMTcxNjk0MDgwMCExNzE3MDI3MjAwLnhtbABtVO2S2yAM/H9P4bkH8EfaJG2Hcn0TDzFyQs8GBnB67dNXmIDlu/uRxKxW0kqsw17e5qm6g/PK6J/PXd0+Vy/8iY0A8iKGV/5UVRV7xDmGWZMPa8SBNS70MwQhRRArSHAludIBnBK1nX505+7cfmm/tR1rNkJOMe7aazEDyWBNATMLZqEmLmfhhjrV8L9oQopnNmqC3gl9LQUieoGrwmHO3en7V1SDMyWEUEBLvso9nA+REM+5aPOuahlmtwVmzaSGv71dLpPyN9jypUGNmv8WV/8PdKglYMmEZYqQr2rmjjXpocDejisafzNouTYaS9iC+Az5DbND4F2cJD5kcDQcrwK/0xifKsarHIzb1Dvzhy7Km8UN0CvLD+2pPhw7/Bzr0xGblwihD2bRITZNDyTyaA53MS24X0li69KUt8arEJ2XhqPIO27c2Yg+QBJZX9nO+AiSLdL5P0jAC96GZkripalR4Wuwd8wdJmOhH52ZuUY/vdXo3mgdGiAZNxASXIJ3ZqCBIuGTtkws4dY78MsUdlr2gxHPEVl7x+XEAYXyOXbG9ayHPSE141Z4H22/HugKqTGbD/JiSjITuq78w/wHs4tey4QEAAA=
--===============4385374180968755410==--

contact-info is null (null is no valid value for setContactInfo method).

Return-Path: <noreply-tls@xanderio.de>
Delivered-To: reports@mailing-services.net
Received: from sogo.saasweb.net
    by sogo.saasweb.net with LMTP
    id xU0WDdHdcGZb1AEArVOHLA
    (envelope-from <noreply-tls@xanderio.de>)
    for <reports@mailing-services.net>; Tue, 18 Jun 2024 03:07:29 +0200
Received: from mail.xanderio.de (carrot.xanderio.de [95.216.142.178])
    (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
     key-exchange X25519 server-signature ECDSA (P-384))
    (No client certificate requested)
    by sogo.saasweb.net (Postfix) with ESMTPS id 7373F4024E
    for <reports@mailing-services.net>; Tue, 18 Jun 2024 03:07:26 +0200 (CEST)
Authentication-Results: sogo.saasweb.net;
    dkim=pass header.d=xanderio.de header.s=202406r header.b=QkJHUOip;
    dkim=pass header.d=xanderio.de header.s=202406e header.b=+gp6k9d8;
    spf=pass (sogo.saasweb.net: domain of noreply-tls@xanderio.de designates 95.216.142.178 as permitted sender) smtp.mailfrom=noreply-tls@xanderio.de;
    dmarc=pass (policy=reject) header.from=xanderio.de
DKIM-Signature: v=1; a=rsa-sha256; s=202406r; d=xanderio.de; c=relaxed/relaxed;
    h=Date:Subject:Message-ID:To:From; t=1718668801; bh=9ZB3H12uu0TMI3J2hLPlvce
    dlLn0ALtxPuyavYgAl9o=; b=QkJHUOip0cyoHoOndPRGV2oVHndEyYnX57Q9foN7QyEZ31GoCc
    NMwuKJTObTmvKDhOUlZLJfmDmWP2tf3lRgMMjiM+vAo2pFZ2AHYMIFCJcb+TLRLPn31me53IMnW
    NZfxeHrKMcSxP1Hvm5eOre1V5Y3k47BaMwg78OuWXxIxl7smmlaRjwmcNAumMEWfjUkL34sBXlB
    Os5ujnj/Pt8NY6MnKIWGqWuXOZ/u0jSNTxIf5QH5nniIRgu/HM9zHxrAfdGXJUGCRkBeSMh1w25
    fK1ePNiJqTef2SCFgpgl84hS7vKSiDMCPKe5Gq/by6lKNTLsT8GpipFSNB3xLjvpQuQ==;
DKIM-Signature: v=1; a=ed25519-sha256; s=202406e; d=xanderio.de; c=relaxed/relaxed;
    h=Date:Subject:Message-ID:To:From; t=1718668801; bh=9ZB3H12uu0TMI3J2hLPlvce
    dlLn0ALtxPuyavYgAl9o=; b=+gp6k9d8T8aeTiUfDPDOXHTgpsnUoS5frD6zS9gNEkHQpnXSxR
    NPWyi1z+GwthkbildEyAMUq6OhdizGq49XAQ==;
From: "TLS Aggregate Report" <noreply-tls@xanderio.de>
To: <reports@mailing-services.net>
Message-ID: <17d9f02b61d65d73.b8f0472e1c0dc903.b47b2552db3a3e76@mail.xanderio.de>
TLS-Report-Domain: mailing-services.net
TLS-Report-Submitter: mail.xanderio.de
Auto-Submitted: auto-generated
Subject: Report Domain: mailing-services.net Submitter: mail.xanderio.de Report-ID:
     <1718582400_5443499956027852877>
Date: Tue, 18 Jun 2024 00:00:01 +0000
Content-Type: multipart/report; report-type="tlsrpt"; 
    boundary="17d9f02b61d68d1b_5727c0e79b584518_b47b2552db3a3e76"
X-Spam-Status: No, score=-0.13
X-Spamd-Bar: /

--17d9f02b61d68d1b_5727c0e79b584518_b47b2552db3a3e76
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

TLS report from mail.xanderio.de

Report Domain: mailing-services.net
Submitter: mail.xanderio.de
Report-ID: 1718582400_5443499956027852877

--17d9f02b61d68d1b_5727c0e79b584518_b47b2552db3a3e76
Content-Type: application/tlsrpt+gzip
Content-Disposition: attachment; 
    filename="mail.xanderio.de!mailing-services.net!1718582400!1718668800.json.gz"
Content-Transfer-Encoding: base64

H4sIAAAAAAAA/22R0W7CMAxFfwXluUFp19LS34CnTQh5jekitQmKXQZD/PucMqZpmxQpjn18b+Jc
VYg9ePcB7ILXHkZUrTqDtxhdWFpUmbLAqCP4XkpXRQyRdcqxm+HCFKU2K53XW2PaeT1LF3r7L9X8
oG6Z6oJn6Fg7fwiq9dMwZCriMYiHs9KX13lTNUVpzL4qy6dyvV5XK1PUTVU0dS0+xzC4ziGp9uV6
P1zSNe+R5ssxuRPTA71o4uh8L7w6YSR5drvYbDenXIgxWGwXjMSJkDOc99BLamXKxpiUObcLCn1Y
EgC94+vSI6vdt7YNIzgvjrINoqEJ48l1SDOX2vVbIE7mf1VkHDSNI8T5BRwYBk1TJ910mCSUPX1S
FyYvCkX2hRzEaor4u25E7lGyyBKlGe1uu9snYO64dfUBAAA=

--17d9f02b61d68d1b_5727c0e79b584518_b47b2552db3a3e76--
antedebaas commented 5 months ago

contact-info is a required field and should never be null. The "TLS-Report-Submitter" value MUST match the value found in the domain [[RFC5321](https://datatracker.ietf.org/doc/html/rfc5321)] of the "contact-info" from the report body. These message header fields MUST be included and should allow for easy searching for all reports submitted by a reporting domain or a particular submitter, https://datatracker.ietf.org/doc/html/rfc8460

antedebaas commented 5 months ago

i checked and even tho its common to provide a message-id some less honorable MTA's don't as its not required. I've added changes in the main branch that will allow for such a case. however the missing contact info breaks the specification and i will thus treat it as an invalid report (which it is).