search

domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
995 stars 213 forks source link

Messages should only be moved after parsing and storing result is succesful #242

Open mback2k opened 3 years ago

mback2k commented 3 years ago

Currently messages are being moved away before the result is stored successfully, e.g. in Elasticsearch. This leads to reports being marked as processed although there may be an error while storing the result.

celya44 commented 1 year ago

hello, +1, and that's a problem. I've lost days of DMARC reports because my ELK was not working and parsedmarc still deleting DMRC reports on mailbox even if they're not stored in elasticsearch :

parsedmarc[696020]:    DEBUG:__init__.py:1087:Found 1 messages in INBOX
parsedmarc[696020]:    DEBUG:__init__.py:1095:Processing 1 messages
parsedmarc[696020]:    DEBUG:__init__.py:1099:Processing message 1 of 1: UID 154
parsedmarc[696020]:     INFO:__init__.py:805:Parsing mail from noreply-dmarc-support@google.com
parsedmarc[696020]:    DEBUG:__init__.py:1139:Deleting message 1 of 1: UID 154
parsedmarc[696020]:     INFO:elastic.py:295:Saving aggregate report to Elasticsearch
parsedmarc[696020]:    ERROR:cli.py:102:Elasticsearch Error: Elasticsearch error: TransportError(503, 'unavailable_shards_exception', '[dmarc_aggregate-2023-07-15][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[dmarc_aggregate-2023-07-15][0]] containing [index {[dmarc_aggregate-2023-07-15][mpYXXokB5z5riAxnqMtq],