Ignore RFC-incompliant date ranges

domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports

https://domainaware.github.io/parsedmarc/

Apache License 2.0

966 stars 210 forks source link

Ignore RFC-incompliant date ranges #282

Open RealSebFox opened 2 years ago

RealSebFox commented 2 years ago

We receive DMARC reports from Trustwave SEG with date ranges exceeding 24 hours. Sometimes they even cover multiple weeks, as in this example:

<date_range>
   <begin>1637392339</begin>
   <end>1639516233</end>
</date_range>

Date ranges excessively exceeding 24-48 hours make data look distorted in Kibana. Since https://datatracker.ietf.org/doc/html/rfc7489#section-7.2 clearly mentions that Visibility comes in the form of daily (or more frequent) Mail, it would probably be best to consider such reports "invalid".

RealSebFox commented 2 years ago

Screenshot from 2022-03-01 23-00-22

Here is an example of the mentioned distortion. I'm going to trace them back manually by exclusion and add filters to exclude the culprits for good. That is a tedious job. Such RFC-incompliant reports keep popping up.

RealSebFox commented 9 months ago

Hello @seanthegeek,

Yesterday, I upgraded from 8.6.1 to 8.6.4 and the problem has returned.

It was broken with this change: https://github.com/domainaware/parsedmarc/blob/master/CHANGELOG.md#863.

Please make sure that invalid or broken reports are not sent to elasticsearch, because they are not RFC compliant and it distorts the timelines!

Thank you!

PS: I'm reverting to 8.6.2