domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
1.01k stars 217 forks source link

IMAP connection errors #311

Closed xbb closed 2 years ago

xbb commented 2 years ago

I've updated to 8.0.2 from a working 7.1.1 install but I'm getting this warning and after a while I believe I get banned from the server:

   INFO:cli.py:667:Starting parsedmarc
  DEBUG:__init__.py:1072:Found 0 messages in dmarc-process
  DEBUG:__init__.py:1080:Processing 0 messages
   INFO:cli.py:885:Watching for email - Quit with ctrl-c
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:70:IMAP connection error. 'IMAPClient' object has no attribute 'fetch_messages'. Reconnecting...
WARNING:imap.py:67:IMAP connection timeout. Reconnecting...
WARNING:imap.py:67:IMAP connection timeout. Reconnecting...
...

Partial config:

[mailbox]
reports_folder = dmarc-process
archive_folder = dmarc-archive
delete = False
watch = True

[imap]
host = ...
user = ...
password = ...
ssl = True

Thank you

seanthegeek commented 2 years ago

I still haven't had a chance to test the possible fix.

@xbb, Can you test it before I push it to pypi by installing the version that is currently in this repo?

You can upgrade to this pre-release version by running

sudo -u parsedmarc -H pip3 install --user -U git+https://github.com/domainaware/parsedmarc.git

xbb commented 2 years ago

I still haven't had a chance to test the possible fix.

@xbb, Can you test it before I push it to pypi by installing the version that is currently in this repo?

You can upgrade to this pre-release version by running

sudo -u parsedmarc -H pip3 install --user -U git+domainaware/parsedmarc.git

I'm getting this error while trying to build a docker image:

#0 8.528 Collecting git+https://github.com/domainaware/parsedmarc.git@master
#0 8.533   Cloning https://github.com/domainaware/parsedmarc.git (to revision master) to /tmp/pip-req-build-nunz8yhn
#0 8.548   Running command git clone --filter=blob:none --quiet https://github.com/domainaware/parsedmarc.git /tmp/pip-req-build-nunz8yhn
#0 11.74   Resolved https://github.com/domainaware/parsedmarc.git to commit af2afddf96fd80c04c1e3ff3d17cfc64c27d75b2
#0 11.75   Preparing metadata (setup.py): started
#0 13.27   Preparing metadata (setup.py): finished with status 'error'
#0 13.34   error: subprocess-exited-with-error
#0 13.34
#0 13.34   × python setup.py egg_info did not run successfully.
#0 13.34   │ exit code: 1
#0 13.34   ╰─> [8 lines of output]
#0 13.34       Traceback (most recent call last):
#0 13.34         File "<string>", line 2, in <module>
#0 13.34         File "<pip-setuptools-caller>", line 34, in <module>
#0 13.34         File "/tmp/pip-req-build-nunz8yhn/setup.py", line 17, in <module>
#0 13.34           from parsedmarc import __version__
#0 13.34         File "/tmp/pip-req-build-nunz8yhn/parsedmarc/__init__.py", line 25, in <module>
#0 13.34           import mailparser
#0 13.34       ModuleNotFoundError: No module named 'mailparser'
#0 13.34       [end of output]
#0 13.34
#0 13.34   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 13.37 error: metadata-generation-failed
#0 13.37
#0 13.37 × Encountered error while generating package metadata.
#0 13.37 ╰─> See above for output.
#0 13.37
#0 13.37 note: This is an issue with the package mentioned above, not pip.
#0 13.37 hint: See above for details.
seanthegeek commented 2 years ago

@xbb I just fixed that. Try again please.

xbb commented 2 years ago

Built fine but now I have this error at runtime:

parsedmarc  | Traceback (most recent call last):
parsedmarc  |   File "/opt/pypy/bin/parsedmarc", line 5, in <module>
parsedmarc  |     from parsedmarc.cli import _main
parsedmarc  |   File "/opt/pypy/lib/pypy3.9/site-packages/parsedmarc/cli.py", line 20, in <module>
parsedmarc  |     from parsedmarc import get_dmarc_reports_from_mailbox, watch_inbox, \
parsedmarc  |   File "<frozen importlib._bootstrap>", line 1058, in _handle_fromlist
parsedmarc  |   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
parsedmarc  |   File "/opt/pypy/lib/pypy3.9/site-packages/parsedmarc/elastic.py", line 6, in <module>
parsedmarc  |     from elasticsearch_dsl.search import Q
parsedmarc  | ModuleNotFoundError: No module named 'elasticsearch_dsl'
parsedmarc exited with code 1
seanthegeek commented 2 years ago

@xbb D'oh! I missed a , in setup.py. That's fixed now. Try one more time please.

xbb commented 2 years ago

@xbb D'oh! I missed a , in setup.py. That's fixed now. Try one more time please.

Thanks! It's up and running now

parsedmarc  |     INFO:cli.py:671:Starting parsedmarc
parsedmarc  |    DEBUG:__init__.py:1072:Found 0 messages in dmarc-process
parsedmarc  |    DEBUG:__init__.py:1080:Processing 0 messages
parsedmarc  |     INFO:cli.py:889:Watching for email - Quit with ctrl-c
parsedmarc  |    DEBUG:__init__.py:1072:Found 0 messages in dmarc-process
parsedmarc  |    DEBUG:__init__.py:1080:Processing 0 messages
parsedmarc  |    DEBUG:__init__.py:1072:Found 0 messages in dmarc-process
parsedmarc  |    DEBUG:__init__.py:1080:Processing 0 messages
parsedmarc  |    DEBUG:__init__.py:1072:Found 1 messages in dmarc-process
parsedmarc  |    DEBUG:__init__.py:1080:Processing 1 messages
parsedmarc  |    DEBUG:__init__.py:1084:Processing message 1 of 1: UID 11
parsedmarc  |     INFO:__init__.py:806:Parsing mail from "xxxxxxxxxxxxxxxxxxxxxxxx"
parsedmarc  |  <xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
parsedmarc  |    DEBUG:__init__.py:1137:Moving aggregate report messages from dmarc-process to dmarc-archive/Aggregate
parsedmarc  |    DEBUG:__init__.py:1144:Moving message 1 of 1: UID 11
parsedmarc  |     INFO:elastic.py:298:Saving aggregate report to Elasticsearch
parsedmarc  | /opt/pypy/lib/pypy3.9/site-packages/dateparser/date_parser.py:35: PytzUsageWarning: The localize method is no longer necessary, as this time zone supports the fold attribute (PEP 495). For more details on migrating to a PEP 495-compliant implementation, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
parsedmarc  |   date_obj = stz.localize(date_obj)
seanthegeek commented 2 years ago

@xbb Can you do one more update and test again? I think I just fixed that warning at the end of the output.

xbb commented 2 years ago

@xbb Can you do one more update and test again? I think I just fixed that warning at the end of the output.

Just tried, the warning is still there