techsneeze / dmarcts-report-parser

A Perl based tool to parse DMARC reports from an IMAP mailbox or from the filesystem, and insert the information into a database. ( Formerly known as imap-dmarcts )
http://www.techsneeze.com/how-parse-dmarc-reports-imap/
GNU General Public License v3.0
223 stars 62 forks source link

Deduplicate reports over (domain, org, reportid), not just over reportid #113

Closed mwander closed 1 year ago

mwander commented 1 year ago

This pull request closes #112.

WARNING: The scheme of a running database must be changed to use the new index definition. Example for MySQL: ALTER TABLE report DROP INDEX domain, ADD UNIQUE KEY domain (domain, org, reportid);

Any idea to automate this?

Without this change, the following failure will occur when encountering an identical (domain, reportid) tuple: DBD::mysql::db do failed: Duplicate entry 'wander.science-wander.science.1677801600.1677888000' for key 'domain' at dmarcts-report-parser.pl line 859. dmarcts-report-parser.pl: aperture-labs.org: wander.science.1677801600.1677888000: Cannot add report to database. Skipped. dmarcts-report-parser.pl: Skipping IMAP message with UID #2869 due to database errors.