domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
1.01k stars 219 forks source link

Parsedmarc crash sometimes #564

Closed davidande closed 1 month ago

davidande commented 1 month ago

hello, here is the log for the crach I have Msgraph connexion Docker Elastic/Kibana 8.13 Docker webtop for Parsedmarc

So it works well, have almost 150 domains + 3 years of logs butt regulary I hace this

Traceback (most recent call last):
  File "/config/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/config/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse
    response.begin()
  File "/usr/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 300, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/config/.local/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/config/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/config/.local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 462, in _make_request
    httplib_response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse
    response.begin()
  File "/usr/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 300, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/config/.local/bin/parsedmarc", line 8, in <module>
    sys.exit(_main())
             ^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/parsedmarc/cli.py", line 1433, in _main
    watch_inbox(
  File "/config/.local/lib/python3.12/site-packages/parsedmarc/__init__.py", line 1668, in watch_inbox
    mailbox_connection.watch(check_callback=check_callback,
  File "/config/.local/lib/python3.12/site-packages/parsedmarc/mail/graph.py", line 219, in watch
    check_callback(self)
  File "/config/.local/lib/python3.12/site-packages/parsedmarc/__init__.py", line 1650, in check_callback
    res = get_dmarc_reports_from_mailbox(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/parsedmarc/__init__.py", line 1451, in get_dmarc_reports_from_mailbox
    messages = connection.fetch_messages(reports_folder, batch_size=batch_size)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/parsedmarc/mail/graph.py", line 150, in fetch_messages
    emails = self._get_all_messages(url, batch_size)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/parsedmarc/mail/graph.py", line 162, in _get_all_messages
    result = self._client.get(url, params=params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/msgraph/core/_graph_client.py", line 46, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/msgraph/core/_graph_client.py", line 93, in get
    return self.graph_session.get(self._graph_url(url), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/msgraph/core/middleware/middleware.py", line 44, in send
    return self._first_middleware.send(request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/msgraph/core/middleware/authorization.py", line 23, in send
    response = super().send(request, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/msgraph/core/middleware/middleware.py", line 66, in send
    return self.next.send(request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/msgraph/core/middleware/retry.py", line 118, in send
    response = super().send(request, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/msgraph/core/middleware/middleware.py", line 66, in send
    return self.next.send(request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/msgraph/core/middleware/telemetry.py", line 22, in send
    response = super().send(request, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/msgraph/core/middleware/middleware.py", line 65, in send
    return super().send(request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/.local/lib/python3.12/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
0it [4:06:12, ?it/s]
abc@webtop:~$ 

any idea?

Thanks David

davidande commented 1 month ago

duplicate