search

domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
1.02k stars 224 forks source link

Some reverse DNS lookups hanging #79

Closed quickwick closed 5 years ago

quickwick commented 5 years ago

Hello,

I'm trying parsedmarc for the first time today, and have run into an issue. It appears to get stuck processing certain messages.

I'm connecting to an Exchange mailbox, via IMAP. In general, I can get it to work. I'll feed it a folder with a few hundred e-mails, it chews through them and generates the .csv and .json files.

However, it appears there are some messages it can't parse. I've let it sit for a couple of hours, no change. I turned on Debug mode, and all I get is this:

DEBUG:__init__.py:1027:Processing message 23 of 414: UID 471

with various values of message x of x: UID x

How can I get more useful information for you on this?

Thanks.

Edit #1 - Additional info on my setup: Windows 10 Python 3.7.3 (fresh install) Exchange 2013 CU 22 I downloaded that GeoLite2-Country.mmdb and put it in C:\GeoIP\ folder as per instructions

Edit #2 - I performed an elimination search, and narrowed the issue down to three report e-mails from google.com, all with large attached reports (26 KB +)

seanthegeek commented 5 years ago

Please forward the original report emails (with the email as an attachment) to dmarc@whalensolutions.com

seanthegeek commented 5 years ago

No response. Closing.

quickwick commented 5 years ago

I sent the original report emails on May 13th. Though I'm realizing now they may not have connected with this issue because I sent them from my work e-mail address, and not my personal e-mail address (the one attached to this github account). Can I re-open this issue, or should I submit a new one?

Google DMARC reports are still failing to parse. I just haven't looked at this for a couple months, as I was waiting to get my DKIM set up before parsing any more DMARC reports.

quickwick commented 5 years ago

I sent the original report e-mails again, this time from the e-mail account associated with this github account.

seanthegeek commented 5 years ago

I just had a chance to look at the samples you sent, and I was able to reproduce the issue. I haven't found the cause yet, but I am working on it.

seanthegeek commented 5 years ago

Good news. I have a fix and I'm including it in the 6.5.0 release in a few days :)

quickwick commented 5 years ago

Great! I look forward to trying it.

seanthegeek commented 5 years ago

@quickwick This has been fixed in parsedmarc 6.5 which was just released. The reports you have are very large, and may still take up to 8 minutes each to process in my testing, but DNS queries should no longer completely hang.

Let me know how it goes either way.

quickwick commented 5 years ago

Works, thanks. I'm not sure exactly how long it look, I had 19 google reports to parse so I just left it running overnight, but it was done in the morning.