search

domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
1.02k stars 224 forks source link

Error pulling dmarc reports #35

Closed DanMcGuinness closed 6 years ago

DanMcGuinness commented 6 years ago

I've installed parsedmarc on a new system (tried both Ubuntu and Debian) and when pulling DMARC reports I'm getting this error:

ERROR:parsedmarc:Elasticsearch Error: Elasticsearch error: Range accepts a single dictionary or a set of keyword arguments.

Error wasn't present in 4.4.1. Hopefully nothing I'm doing on my new system...

DanMcGuinness commented 6 years ago

Tested some more and installed 4.4.1 on my new system. Same error... So must be something in my OS build. Any help appreciated!

seanthegeek commented 6 years ago

Hi,

Weird. Please run it with the --debug option and post the output.

maxsalov commented 6 years ago

Hi,

I have the same issue. Here is debug output

Nov 23 05:18:46 srl-proxy-02 systemd[1]: Started parsedmarc mailbox watcher.
Nov 23 05:18:48 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:18:48 srl-proxy-02 parsedmarc[89609]: INFO:parsedmarc:Watching for email - Quit with ctrl-c
Nov 23 05:18:48 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:19:18 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:19:18 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:19:48 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:19:49 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:20:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:20:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:20:49 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:20:49 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:21:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:21:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:21:49 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:21:49 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:22:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:22:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:22:49 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:22:49 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:23:18 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:23:18 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 2 messages in IMAP folder INBOX
Nov 23 05:23:18 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Processing message 1 of 2: UID 17
Nov 23 05:23:18 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Processing message 2 of 2: UID 18
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Moving aggregate report messages from INBOX to INBOX
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Moving message 1 of 2: UID 17
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Moving message UID(s) 17 to Archive/Aggregate
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Moving message 2 of 2: UID 18
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Moving message UID(s) 18 to Archive/Aggregate
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:IMAP server supports: ['IMAP4', 'IMAP4REV1', 'AUTH=PLAIN', 'AUTH=NTLM', 'AUTH=GSSAPI', 'SASL-IR', 'UIDPLUS', 'MOVE', 'ID', 'UNSELECT', 'XPROXY3', 'CLIENTNETWORKPRESENCELOCATION', 'CHILDREN', 'IDLE', 'NAMESPACE', 'LITERAL+']
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Found 0 messages in IMAP folder INBOX
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Saving aggregate report to Elasticsearch
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: ERROR:parsedmarc:Elasticsearch Error: Elasticsearch error: Range accepts a single dictionary or a set of keyword arguments.
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: DEBUG:parsedmarc:Saving aggregate report to Elasticsearch
Nov 23 05:23:19 srl-proxy-02 parsedmarc[89609]: ERROR:parsedmarc:Elasticsearch Error: Elasticsearch error: Range accepts a single dictionary or a set of keyword arguments.
maxsalov commented 6 years ago

I have to add that parsedmarc is installed according to the instruction on Debian 9

DanMcGuinness commented 6 years ago

Debug logs here show the same thing. I used a VM installed using Debian from Parallels previously and that's working fine. The problems started when I tried Ubuntu or Debian using Azure builds. Happens both in 'normal' and when using PyPy.

JeroenvO commented 6 years ago

Same issue on debian 9 in LXC, when installed in global python3 cpython.

JeroenvO commented 6 years ago

On all reports, its a production system so i cannot send the reports here. The parsing to JSON is correct, the problem is when sending it to ElasticSearch. Reports are also parsed with postmarkapp, which shows correct results.

maxsalov commented 6 years ago

/opt/venvs/parsedmarc/bin/pip3 install elasticsearch-dsl==6.2.1 fixes error. Version 6.3.0 is installed by default.

seanthegeek commented 6 years ago

@maxsalov Thank you!!! I've released parsedmarc 5.0.1, which statically sets the requirement elasticsearch-dsl==6.2.1. Everyone please upgrade to this version.

I'm looking into making forwards compatible code later.

seanthegeek commented 6 years ago

Upstream issue https://github.com/elastic/elasticsearch-dsl-py/issues/1074

seanthegeek commented 6 years ago

D'oh! found and fixed another bug. Please upgrade to parsedmarc==5.0.2