search

domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
1.01k stars 219 forks source link

Ignore RFC-incompliant date ranges #282

Closed RealSebFox closed 2 weeks ago

RealSebFox commented 2 years ago

We receive DMARC reports from Trustwave SEG with date ranges exceeding 24 hours. Sometimes they even cover multiple weeks, as in this example:

<date_range>
   <begin>1637392339</begin>
   <end>1639516233</end>
</date_range>

Date ranges excessively exceeding 24-48 hours make data look distorted in Kibana. Since https://datatracker.ietf.org/doc/html/rfc7489#section-7.2 clearly mentions that Visibility comes in the form of daily (or more frequent) Mail, it would probably be best to consider such reports "invalid".

RealSebFox commented 2 years ago

Screenshot from 2022-03-01 23-00-22

Here is an example of the mentioned distortion. I'm going to trace them back manually by exclusion and add filters to exclude the culprits for good. That is a tedious job. Such RFC-incompliant reports keep popping up.

RealSebFox commented 1 year ago

Hello @seanthegeek,

Yesterday, I upgraded from 8.6.1 to 8.6.4 and the problem has returned.

It was broken with this change: https://github.com/domainaware/parsedmarc/blob/master/CHANGELOG.md#863.

Please make sure that invalid or broken reports are not sent to elasticsearch, because they are not RFC compliant and it distorts the timelines!

Thank you!

PS: I'm reverting to 8.6.2

PHPGangsta commented 3 weeks ago

Hello,

For me the problem is also back, graphs in Kibana are broken again after 8.6.3 because invalid reports get imported to ElasticSearch again.

https://github.com/domainaware/parsedmarc/commit/1655b84cc28202de2f6989939269ae182a0ebfdf#diff-9234afbce8b159da96ee1a50bdca3307fc6f2cfbb7ac00f1fd41120c5a0d1a10R278

@seanthegeek Can you please explain why the Exception has been changed to only an error? Is it possible to have an error and DON'T import the report to ElasticSearch? I don't care if it's an error or an exception, important is that invalid reports don't get imported into ElasticSearch.

Here a screenshot with broken graphs in Kibana. I selected "Last 1 month", but because some invalid reports did contain data of multiple month in a single report, it does not only show 1 month, but a graph starting in February: grafik

seanthegeek commented 2 weeks ago

Fixed in 8.15.4. Sorry about that.

RealSebFox commented 2 weeks ago

Awesome. Thank you, @seanthegeek!

PHPGangsta commented 2 weeks ago

Thank you @seanthegeek ! I didn't try the new code (yet), but just from checking the new code I would say that it should be "_error" instead of "error" in line 522, so the correct error message is given to InvalidAggregateReport? Or you rename the variable one line above to "error".

https://github.com/domainaware/parsedmarc/commit/52ccf0536c620c2ae144949f0a5312826477b3ef#r148355094

seanthegeek commented 2 weeks ago

Yep. I fixed that a few minutes after that commit. Oops.

PHPGangsta commented 2 weeks ago

Yep. I fixed that a few minutes after that commit. Oops.

Ah, yes, I see. Thanks! 👍