IMAP watch not working (No Output / Lines duplicated)

[w4tzmann]

As soon as I set IMAP watch to true, the output no longer works as expected. Either there is no entry at all (except for the headline), or the entries are saved several times in the CSV/JSON file.

parsedmarc -v 8.6.0

parsedmarc.ini

[general] save_aggregate = True save_forensic = False output = /output silent = False debug = true

[imap] host = redacted user = redacted password = redacted

[mailbox] watch = True delete = False reports_folder = Inbox

First run with 1 mail in Inbox:

Log:

2023-05-17 20:15:00,870 - INFO - [cli.py:799] - Starting parsedmarc 2023-05-17 20:15:01,387 - DEBUG - [init.py:1087] - Found 1 messages in Inbox 2023-05-17 20:15:01,388 - DEBUG - [init.py:1095] - Processing 1 messages 2023-05-17 20:15:01,388 - DEBUG - [init.py:1099] - Processing message 1 of 1: UID 7 2023-05-17 20:15:01,425 - INFO - [init.py:805] - Parsing mail from admin 2023-05-17 20:15:01,913 - DEBUG - [init.py:1152] - Moving aggregate report messages from Inbox to Archive/Aggregate 2023-05-17 20:15:01,914 - DEBUG - [init.py:1159] - Moving message 1 of 1: UID 7 2023-05-17 20:15:02,056 - INFO - [cli.py:1054] - Watching for email - Quit with ctrl-c 2023-05-17 20:15:02,712 - DEBUG - [init.py:1087] - Found 0 messages in Inbox 2023-05-17 20:15:02,713 - DEBUG - [init.py:1095] - Processing 0 messages 2023-05-17 20:15:28,308 - DEBUG - [init.py:1087] - Found 0 messages in Inbox 2023-05-17 20:15:28,308 - DEBUG - [init.py:1095] - Processing 0 messages

cli:

parsedmarc --debug --verbose -c parsedmarc.ini --log-file parsedmarc.log INFO:cli.py:799:Starting parsedmarc 0it [00:00, ?it/s] DEBUG:init.py:1087:Found 1 messages in Inbox DEBUG:init.py:1095:Processing 1 messages DEBUG:init.py:1099:Processing message 1 of 1: UID 7 INFO:init.py:805:Parsing mail from admin DEBUG:init.py:1152:Moving aggregate report messages from Inbox to Archive/Aggregate DEBUG:init.py:1159:Moving message 1 of 1: UID 7 { "aggregate_reports": [ { "xml_schema": "draft", "report_metadata": { "org_name": "google.com", "org_email": "noreply-dmarc-support@google.com", "org_extra_contact_info": "https://support.google.com/a/answer/2466580", "report_id": "redacted", "begin_date": "2023-04-24 00:00:00", "end_date": "2023-04-24 23:59:59", "errors": [] }, "policy_published": { "domain": "redacted", "adkim": "r", "aspf": "r", "p": "none", "sp": "none", "pct": "100", "fo": "0" }, "records": [ { "source": { "ip_address": "redacted", "country": "US", "reverse_dns": "redactec", "base_domain": "redacted" }, "count": 1, "alignment": { "spf": false, "dkim": false, "dmarc": false }, "policy_evaluated": { "disposition": "none", "dkim": "fail", "spf": "fail", "policy_override_reasons": [] }, "identifiers": { "header_from": "redacted", "envelope_from": "redacted", "envelope_to": null }, "auth_results": { "dkim": [ { "domain": "redacted", "selector": "key2", "result": "fail" }, { "domain": "redacted", "selector": "dkim-mc", "result": "fail" } ], "spf": [ { "domain": "redacted", "scope": "mfrom", "result": "pass" } ] } } ] } ], "forensic_reports": [] }

INFO:cli.py:1054:Watching for email - Quit with ctrl-c

DEBUG:init.py:1087:Found 0 messages in Inbox DEBUG:init.py:1095:Processing 0 messages { "aggregate_reports": [], "forensic_reports": [] }

aggregate.json:

[ { "xml_schema": "draft", "report_metadata": { "org_name": "google.com", "org_email": "noreply-dmarc-support@google.com", "org_extra_contact_info": "https://support.google.com/a/answer/2466580", "report_id": "redacted", "begin_date": "2023-04-24 00:00:00", "end_date": "2023-04-24 23:59:59", "errors": [] }, "policy_published": { "domain": "redacted", "adkim": "r", "aspf": "r", "p": "none", "sp": "none", "pct": "100", "fo": "0" }, "records": [ { "source": { "ip_address": "redacted", "country": "US", "reverse_dns": "redacted", "base_domain": "redacted" }, "count": 1, "alignment": { "spf": false, "dkim": false, "dmarc": false }, "policy_evaluated": { "disposition": "none", "dkim": "fail", "spf": "fail", "policy_override_reasons": [] }, "identifiers": { "header_from": "redacted", "envelope_from": "redacted", "envelope_to": null }, "auth_results": { "dkim": [ { "domain": "redacted", "selector": "key2", "result": "fail" }, { "domain": "redacted", "selector": "dkim-mc", "result": "fail" } ], "spf": [ { "domain": "redacted", "scope": "mfrom", "result": "pass" } ] } } ] } ], { "xml_schema": "draft", "report_metadata": { "org_name": "google.com", "org_email": "noreply-dmarc-support@google.com", "org_extra_contact_info": "https://support.google.com/a/answer/2466580", "report_id": "redacted", "begin_date": "2023-04-24 00:00:00", "end_date": "2023-04-24 23:59:59", "errors": [] }, "policy_published": { "domain": "redacted", "adkim": "r", "aspf": "r", "p": "none", "sp": "none", "pct": "100", "fo": "0" }, "records": [ { "source": { "ip_address": "redacted", "country": "US", "reverse_dns": "redacted", "base_domain": "redacted" }, "count": 1, "alignment": { "spf": false, "dkim": false, "dmarc": false }, "policy_evaluated": { "disposition": "none", "dkim": "fail", "spf": "fail", "policy_override_reasons": [] }, "identifiers": { "header_from": "redacted", "envelope_from": "redacted", "envelope_to": null }, "auth_results": { "dkim": [ { "domain": "redacted", "selector": "key2", "result": "fail" }, { "domain": "redacted", "selector": "dkim-mc", "result": "fail" } ], "spf": [ { "domain": "redacted", "scope": "mfrom", "result": "pass" } ] } } ] } ], { "xml_schema": "draft", "report_metadata": { "org_name": "google.com", "org_email": "noreply-dmarc-support@google.com", "org_extra_contact_info": "https://support.google.com/a/answer/2466580", "report_id": "redacted", "begin_date": "2023-04-24 00:00:00", "end_date": "2023-04-24 23:59:59", "errors": [] }, "policy_published": { "domain": "redacted", "adkim": "r", "aspf": "r", "p": "none", "sp": "none", "pct": "100", "fo": "0" }, "records": [ { "source": { "ip_address": "redacted", "country": "US", "reverse_dns": "redacted", "base_domain": "redacted" }, "count": 1, "alignment": { "spf": false, "dkim": false, "dmarc": false }, "policy_evaluated": { "disposition": "none", "dkim": "fail", "spf": "fail", "policy_override_reasons": [] }, "identifiers": { "header_from": "redacted", "envelope_from": "redacted", "envelope_to": null }, "auth_results": { "dkim": [ { "domain": "redacted", "selector": "key2", "result": "fail" }, { "domain": "redacted", "selector": "dkim-mc", "result": "fail" } ], "spf": [ { "domain": "redacted", "scope": "mfrom", "result": "pass" } ] } } ] } ]

aggregate.csv:

xml_schema,org_name,org_email,org_extra_contact_info,report_id,begin_date,end_date,errors,domain,adkim,aspf,p,sp,pct,fo,source_ip_address,source_country,source_reverse_dns,source_base_domain,count,spf_aligned,dkim_aligned,dmarc_aligned,disposition,policy_override_reasons,policy_override_comments,envelope_from,header_from,envelope_to,dkim_domains,dkim_selectors,dkim_results,spf_domains,spf_scopes,spf_results draft,google.com,noreply-dmarc-support@google.com,https://support.google.com/a/answer/2466580,redacted,2023-04-24 00:00:00,2023-04-24 23:59:59,,redacted,r,r,none,none,100,0,redacted,US,redacted,redacted,1,False,False,False,none,,,redacted,redacted,,"redacted,redacted","key2,dkim-mc","fail,fail",redacted,mfrom,pass draft,google.com,noreply-dmarc-support@google.com,https://support.google.com/a/answer/2466580,redacted,2023-04-24 00:00:00,2023-04-24 23:59:59,,redacted,r,r,none,none,100,0,redacted,US,redacted,redacted,1,False,False,False,none,,,redacted,redacted,,"redacted,redacted","key2,dkim-mc","fail,fail",redacted,mfrom,pass draft,google.com,noreply-dmarc-support@google.com,https://support.google.com/a/answer/2466580,redacted,2023-04-24 00:00:00,2023-04-24 23:59:59,,redacted,r,r,none,none,100,0,redacted,US,redacted,redacted,1,False,False,False,none,,,redacted,redacted,,"redacted,redacted","key2,dkim-mc","fail,fail",redacted,mfrom,pass

second run started without mail in inbox and adding it afterwards:

Log:

2023-05-17 20:29:15,931 - INFO - [cli.py:799] - Starting parsedmarc 2023-05-17 20:29:16,379 - DEBUG - [init.py:1087] - Found 0 messages in Inbox 2023-05-17 20:29:16,379 - DEBUG - [init.py:1095] - Processing 0 messages 2023-05-17 20:29:16,429 - INFO - [cli.py:1054] - Watching for email - Quit with ctrl-c 2023-05-17 20:29:16,791 - DEBUG - [init.py:1087] - Found 0 messages in Inbox 2023-05-17 20:29:16,791 - DEBUG - [init.py:1095] - Processing 0 messages 2023-05-17 20:29:39,081 - DEBUG - [init.py:1087] - Found 1 messages in Inbox 2023-05-17 20:29:39,081 - DEBUG - [init.py:1095] - Processing 1 messages 2023-05-17 20:29:39,081 - DEBUG - [init.py:1099] - Processing message 1 of 1: UID 8 2023-05-17 20:29:39,115 - INFO - [init.py:805] - Parsing mail from admin 2023-05-17 20:29:39,469 - DEBUG - [init.py:1152] - Moving aggregate report messages from Inbox to Archive/Aggregate 2023-05-17 20:29:39,469 - DEBUG - [init.py:1159] - Moving message 1 of 1: UID 8 2023-05-17 20:30:09,735 - DEBUG - [init.py:1087] - Found 0 messages in Inbox 2023-05-17 20:30:09,735 - DEBUG - [init.py:1095] - Processing 0 messages

cli:

parsedmarc --debug --verbose -c parsedmarc.ini --log-file /output/parsedmarc.log INFO:cli.py:799:Starting parsedmarc 0it [00:00, ?it/s] DEBUG:init.py:1087:Found 0 messages in Inbox DEBUG:init.py:1095:Processing 0 messages { "aggregate_reports": [], "forensic_reports": [] }

INFO:cli.py:1054:Watching for email - Quit with ctrl-c

DEBUG:init.py:1087:Found 0 messages in Inbox DEBUG:init.py:1095:Processing 0 messages { "aggregate_reports": [], "forensic_reports": [] }

DEBUG:init.py:1087:Found 1 messages in Inbox DEBUG:init.py:1095:Processing 1 messages DEBUG:init.py:1099:Processing message 1 of 1: UID 8 INFO:init.py:805:Parsing mail from admin DEBUG:init.py:1152:Moving aggregate report messages from Inbox to Archive/Aggregate DEBUG:init.py:1159:Moving message 1 of 1: UID 8 { "aggregate_reports": [ { "xml_schema": "draft", "report_metadata": { "org_name": "google.com", "org_email": "noreply-dmarc-support@google.com", "org_extra_contact_info": "https://support.google.com/a/answer/2466580", "report_id": "redacted", "begin_date": "2023-04-24 00:00:00", "end_date": "2023-04-24 23:59:59", "errors": [] }, "policy_published": { "domain": "redacted", "adkim": "r", "aspf": "r", "p": "none", "sp": "none", "pct": "100", "fo": "0" }, "records": [ { "source": { "ip_address": "redacted", "country": "US", "reverse_dns": "redacted", "base_domain": "redacted" }, "count": 1, "alignment": { "spf": false, "dkim": false, "dmarc": false }, "policy_evaluated": { "disposition": "none", "dkim": "fail", "spf": "fail", "policy_override_reasons": [] }, "identifiers": { "header_from": "redacted", "envelope_from": "redacted", "envelope_to": null }, "auth_results": { "dkim": [ { "domain": "redacted", "selector": "key2", "result": "fail" }, { "domain": "redacted", "selector": "dkim-mc", "result": "fail" } ], "spf": [ { "domain": "redacted", "scope": "mfrom", "result": "pass" } ] } } ] } ], "forensic_reports": [] }

DEBUG:init.py:1087:Found 0 messages in Inbox DEBUG:init.py:1095:Processing 0 messages { "aggregate_reports": [], "forensic_reports": [] }

aggregate.csv:

xml_schema,org_name,org_email,org_extra_contact_info,report_id,begin_date,end_date,errors,domain,adkim,aspf,p,sp,pct,fo,source_ip_address,source_country,source_reverse_dns,source_base_domain,count,spf_aligned,dkim_aligned,dmarc_aligned,disposition,policy_override_reasons,policy_override_comments,envelope_from,header_from,envelope_to,dkim_domains,dkim_selectors,dkim_results,spf_domains,spf_scopes,spf_results

aggregate.json:

[]

third run with watch = False in parsedmarc.ini: log:

2023-05-17 20:36:41,280 - INFO - [cli.py:799] - Starting parsedmarc 2023-05-17 20:36:41,759 - DEBUG - [init.py:1087] - Found 1 messages in Inbox 2023-05-17 20:36:41,760 - DEBUG - [init.py:1095] - Processing 1 messages 2023-05-17 20:36:41,761 - DEBUG - [init.py:1099] - Processing message 1 of 1: UID 9 2023-05-17 20:36:41,784 - INFO - [init.py:805] - Parsing mail from admin 2023-05-17 20:36:42,221 - DEBUG - [init.py:1152] - Moving aggregate report messages from Inbox to Archive/Aggregate 2023-05-17 20:36:42,222 - DEBUG - [init.py:1159] - Moving message 1 of 1: UID 9

cli:

parsedmarc --debug --verbose -c parsedmarc.ini --log-file /output/parsedmarc.log INFO:cli.py:799:Starting parsedmarc 0it [00:00, ?it/s] DEBUG:init.py:1087:Found 1 messages in Inbox DEBUG:init.py:1095:Processing 1 messages DEBUG:init.py:1099:Processing message 1 of 1: UID 9 INFO:init.py:805:Parsing mail from admin DEBUG:init.py:1152:Moving aggregate report messages from Inbox to Archive/Aggregate DEBUG:init.py:1159:Moving message 1 of 1: UID 9 { "aggregate_reports": [ { "xml_schema": "draft", "report_metadata": { "org_name": "google.com", "org_email": "noreply-dmarc-support@google.com", "org_extra_contact_info": "https://support.google.com/a/answer/2466580", "report_id": "redacted", "begin_date": "2023-04-24 00:00:00", "end_date": "2023-04-24 23:59:59", "errors": [] }, "policy_published": { "domain": "redacted", "adkim": "r", "aspf": "r", "p": "none", "sp": "none", "pct": "100", "fo": "0" }, "records": [ { "source": { "ip_address": "redacted", "country": "US", "reverse_dns": "redacted", "base_domain": "redacted" }, "count": 1, "alignment": { "spf": false, "dkim": false, "dmarc": false }, "policy_evaluated": { "disposition": "none", "dkim": "fail", "spf": "fail", "policy_override_reasons": [] }, "identifiers": { "header_from": "redacted", "envelope_from": "redacted", "envelope_to": null }, "auth_results": { "dkim": [ { "domain": "redacted", "selector": "key2", "result": "fail" }, { "domain": "redacted", "selector": "dkim-mc", "result": "fail" } ], "spf": [ { "domain": "redacted", "scope": "mfrom", "result": "pass" } ] } } ] } ], "forensic_reports": [] }

aggregate.json:

[ { "xml_schema": "draft", "report_metadata": { "org_name": "google.com", "org_email": "noreply-dmarc-support@google.com", "org_extra_contact_info": "https://support.google.com/a/answer/2466580", "report_id": "redacted", "begin_date": "2023-04-24 00:00:00", "end_date": "2023-04-24 23:59:59", "errors": [] }, "policy_published": { "domain": "redacted", "adkim": "r", "aspf": "r", "p": "none", "sp": "none", "pct": "100", "fo": "0" }, "records": [ { "source": { "ip_address": "redacted", "country": "US", "reverse_dns": "redacted", "base_domain": "redacted" }, "count": 1, "alignment": { "spf": false, "dkim": false, "dmarc": false }, "policy_evaluated": { "disposition": "none", "dkim": "fail", "spf": "fail", "policy_override_reasons": [] }, "identifiers": { "header_from": "redacted", "envelope_from": "redacted", "envelope_to": null }, "auth_results": { "dkim": [ { "domain": "redacted", "selector": "key2", "result": "fail" }, { "domain": "redacted", "selector": "dkim-mc", "result": "fail" } ], "spf": [ { "domain": "redacted", "scope": "mfrom", "result": "pass" } ] } } ] } ]

aggregate.csv:

xml_schema,org_name,org_email,org_extra_contact_info,report_id,begin_date,end_date,errors,domain,adkim,aspf,p,sp,pct,fo,source_ip_address,source_country,source_reverse_dns,source_base_domain,count,spf_aligned,dkim_aligned,dmarc_aligned,disposition,policy_override_reasons,policy_override_comments,envelope_from,header_from,envelope_to,dkim_domains,dkim_selectors,dkim_results,spf_domains,spf_scopes,spf_results draft,google.com,noreply-dmarc-support@google.com,https://support.google.com/a/answer/2466580,redacted,2023-04-24 00:00:00,2023-04-24 23:59:59,,redacted,r,r,none,none,100,0,redacted,US,redacted,redacted,1,False,False,False,none,,,redacted,redacted,,"redacted,redacted","key2,dkim-mc","fail,fail",redacted,mfrom,pass

Dockerfile:

FROM python:alpine

RUN apk add --update --no-cache libxml2-dev libxslt-dev
RUN apk add --update --no-cache --virtual .build_deps build-base libffi-dev \
    && pip install parsedmarc \
    && apk del .build_deps

Has anyone else faced this issue and could solve this?