search

domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
962 stars 210 forks source link

ModuleNotFoundError: No module named 'msgraph' #464

Closed laughk closed 4 months ago

laughk commented 5 months ago

Description

I have installed parsedmarc but it does not work with the following error:

$ parsedmarc --help
Traceback (most recent call last):
  File "/tmp/parsedmarc/bin/parsedmarc", line 5, in <module>
    from parsedmarc.cli import _main
  File "/tmp/parsedmarc/lib/python3.11/site-packages/parsedmarc/__init__.py", line 31, in <module>
    from parsedmarc.mail import MailboxConnection
  File "/tmp/parsedmarc/lib/python3.11/site-packages/parsedmarc/mail/__init__.py", line 2, in <module>
    from parsedmarc.mail.graph import MSGraphConnection
  File "/tmp/parsedmarc/lib/python3.11/site-packages/parsedmarc/mail/graph.py", line 10, in <module>
    from msgraph.core import GraphClient
ModuleNotFoundError: No module named 'msgraph'

I am seeing this error in at least the following environments

Steps to reproduce

This occurs when you install parsedmarc and run the command in the following flow.

$ python -m venv venv
$ source venv/bin/activate
(venv) $ python -m pip install parsedmarc

Running parsedmarc in this state will result in the following

$ parsedmarc
Traceback (most recent call last):
  File "/tmp/parsedmarc/bin/parsedmarc", line 5, in <module>
    from parsedmarc.cli import _main
  File "/tmp/parsedmarc/lib/python3.11/site-packages/parsedmarc/__init__.py", line 31, in <module>
    from parsedmarc.mail import MailboxConnection
  File "/tmp/parsedmarc/lib/python3.11/site-packages/parsedmarc/mail/__init__.py", line 2, in <module>
    from parsedmarc.mail.graph import MSGraphConnection
  File "/tmp/parsedmarc/lib/python3.11/site-packages/parsedmarc/mail/graph.py", line 10, in <module>
    from msgraph.core import GraphClient
ModuleNotFoundError: No module named 'msgraph'

What else I have investigated

I looked up the module name msgraph and found that it refers to the dependent package msgraph-core, The latest version of msgraph-core, 1.0.0, was released on 2024-01-23, the first major release since 2021-07-27.

On the other hand, if you look at project.toml, you will see that >= 0.2.2.

https://github.com/domainaware/parsedmarc/blob/7d2b431e5f20bdcdb330c4fbb23ce7df5fb0642f/pyproject.toml#L50

Therefore, it is assumed that a new installation of parsedmarc installs msgraph-core 1.0.0, and that it no longer works.

Workaround

It seems to be possible to avoid this problem by specifying a msgraph-core version less than 1.0.0 as follows.

$ python -m venv venv
$ source venv/bin/activate
(venv) $ python -m pip install parsedmarc msgraph-core==0.2.2
(venv) $ parsedmarc --help
usage: parsedmarc [-h] [-c CONFIG_FILE] [--strip-attachment-payloads] [-o OUTPUT] [--aggregate-json-filename AGGREGATE_JSON_FILENAME] [--forensic-json-filename FORENSIC_JSON_FILENAME]
                  [--aggregate-csv-filename AGGREGATE_CSV_FILENAME] [--forensic-csv-filename FORENSIC_CSV_FILENAME] [-n NAMESERVERS [NAMESERVERS ...]] [-t DNS_TIMEOUT] [--offline]
                  [-s] [-w] [--verbose] [--debug] [--log-file LOG_FILE] [-v]
                  [file_path ...]

Parses DMARC reports

positional arguments:
  file_path             one or more paths to aggregate or forensic report files, emails, or mbox files'

options:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        a path to a configuration file (--silent implied)
  --strip-attachment-payloads
                        remove attachment payloads from forensic report output
  -o OUTPUT, --output OUTPUT
                        write output files to the given directory
  --aggregate-json-filename AGGREGATE_JSON_FILENAME
                        filename for the aggregate JSON output file
  --forensic-json-filename FORENSIC_JSON_FILENAME
                        filename for the forensic JSON output file
  --aggregate-csv-filename AGGREGATE_CSV_FILENAME
                        filename for the aggregate CSV output file
  --forensic-csv-filename FORENSIC_CSV_FILENAME
                        filename for the forensic CSV output file
  -n NAMESERVERS [NAMESERVERS ...], --nameservers NAMESERVERS [NAMESERVERS ...]
                        nameservers to query
  -t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT
                        number of seconds to wait for an answer from DNS (default: 2.0)
  --offline             do not make online queries for geolocation or DNS
  -s, --silent          only print errors
  -w, --warnings        print warnings in addition to errors
  --verbose             more verbose output
  --debug               print debugging information
  --log-file LOG_FILE   output logging to a file
  -v, --version         show program's version number and exit
jose-m-ferreira commented 5 months ago

confirm workaround works: ignore my tasteless non-usage of venv please.

jose@ubuntudmarc:~$ pip3 show -f parsedmarc Name: parsedmarc Version: 8.6.4 Summary: A Python package and CLI for parsing aggregate and forensic DMARC reports Home-page: Author: Author-email: Sean Whalen whalenster@gmail.com License: Location: /usr/local/lib/python3.11/dist-packages Requires: azure-identity, azure-monitor-ingestion, boto3, dateparser, dnspython, elasticsearch, elasticsearch-dsl, expiringdict, geoip2, google-api-core, google-api-python-client, google-auth, google-auth-httplib2, google-auth-oauthlib, imapclient, kafka-python, lxml, mailsuite, msgraph-core, publicsuffixlist, requests, tqdm, urllib3, xmltodict Required-by: Files: ../../../bin/parsedmarc parsedmarc-8.6.4.dist-info/INSTALLER parsedmarc-8.6.4.dist-info/METADATA parsedmarc-8.6.4.dist-info/RECORD parsedmarc-8.6.4.dist-info/REQUESTED parsedmarc-8.6.4.dist-info/WHEEL parsedmarc-8.6.4.dist-info/entry_points.txt parsedmarc-8.6.4.dist-info/licenses/LICENSE parsedmarc/init.py parsedmarc/pycache/init.cpython-311.pyc parsedmarc/pycache/cli.cpython-311.pyc parsedmarc/pycache/elastic.cpython-311.pyc parsedmarc/pycache/kafkaclient.cpython-311.pyc parsedmarc/pycache/log.cpython-311.pyc parsedmarc/pycache/loganalytics.cpython-311.pyc parsedmarc/pycache/s3.cpython-311.pyc parsedmarc/pycache/splunk.cpython-311.pyc parsedmarc/pycache/syslog.cpython-311.pyc parsedmarc/pycache/utils.cpython-311.pyc parsedmarc/cli.py parsedmarc/elastic.py parsedmarc/kafkaclient.py parsedmarc/log.py parsedmarc/loganalytics.py parsedmarc/mail/init.py parsedmarc/mail/pycache/init.cpython-311.pyc parsedmarc/mail/pycache/gmail.cpython-311.pyc parsedmarc/mail/pycache/graph.cpython-311.pyc parsedmarc/mail/pycache/imap.cpython-311.pyc parsedmarc/mail/pycache/mailbox_connection.cpython-311.pyc parsedmarc/mail/gmail.py parsedmarc/mail/graph.py parsedmarc/mail/imap.py parsedmarc/mail/mailbox_connection.py parsedmarc/resources/init.py parsedmarc/resources/pycache/init.cpython-311.pyc parsedmarc/resources/dbip/README.md parsedmarc/resources/dbip/init.py parsedmarc/resources/dbip/pycache/init.cpython-311.pyc parsedmarc/resources/dbip/dbip-country-lite.mmdb parsedmarc/s3.py parsedmarc/splunk.py parsedmarc/syslog.py parsedmarc/utils.py

jose@ubuntudmarc:~$ parsedmarc --help Traceback (most recent call last): File "/usr/local/bin/parsedmarc", line 5, in from parsedmarc.cli import _main File "/usr/local/lib/python3.11/dist-packages/parsedmarc/init.py", line 31, in from parsedmarc.mail import MailboxConnection File "/usr/local/lib/python3.11/dist-packages/parsedmarc/mail/init.py", line 2, in from parsedmarc.mail.graph import MSGraphConnection File "/usr/local/lib/python3.11/dist-packages/parsedmarc/mail/graph.py", line 10, in from msgraph.core import GraphClient ModuleNotFoundError: No module named 'msgraph'

jose@ubuntudmarc:~$ sudo pip3 uninstall msgraph-core --break-system-package Found existing installation: msgraph-core 1.0.0 Uninstalling msgraph-core-1.0.0: Would remove: /usr/local/lib/python3.11/dist-packages/msgraph_core-1.0.0.dist-info/ /usr/local/lib/python3.11/dist-packages/msgraph_core/ Proceed (Y/n)? y Successfully uninstalled msgraph-core-1.0.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

jose@ubuntudmarc:~$ sudo pip3 install msgraph-core==0.2.2 --break-system-packages Collecting msgraph-core==0.2.2 Downloading msgraph_core-0.2.2-py3-none-any.whl (15 kB) Installing collected packages: msgraph-core Successfully installed msgraph-core-0.2.2 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

jose@ubuntudmarc:~$ parsedmarc --version 8.6.4

tjillez commented 5 months ago

I did all this @jose-m-ferreira but i'm still getting:

parsedmarc.service: Failed with result 'exit-code'. parsedmarc.service: Main process exited, code=exited, status=1/FAILURE ModuleNotFoundError: No module named 'msgraph' from msgraph.core import GraphClient File "/opt/parsedmarc/venv/lib/python3.10/site-packages/parsedmarc/mail/graph.py", line 10, in from parsedmarc.mail.graph import MSGraphConnection JaFile "/opt/parsedmarc/venv/lib/python3.10/site-packages/parsedmarc/mail/init.py", line 2, in from parsedmarc.mail import MailboxConnection File "/opt/parsedmarc/venv/lib/python3.10/site-packages/parsedmarc/init.py", line 31, in Jafrom parsedmarc.cli import _main File "/opt/parsedmarc/venv/bin/parsedmarc", line 5, in

pip3 list | grep msgraph msgraph-core 0.2.2

jose-m-ferreira commented 5 months ago

@tjilez

looks like you have msgraph-core 0.2.2 alright, right, but i'm curious as if any would be in the venv environment at /opt/parsedmarc and if so what version.

tjillez commented 5 months ago

@jose-m-ferreira
(venv) root@philip:/home/philip# parsedmarc --version 8.6.4

I think i got it working now :)

But i get: ERROR:cli.py:799:You must supply input files or a mailbox connection. Do you know where i should put input files?

jose-m-ferreira commented 5 months ago

you have to configure a parsedmarc.ini somewhere. i actually haven't been able to get imap out of gmail to work.

laughk commented 4 months ago

This issue has been resolved in the following commit (version 8.7.0).

https://github.com/domainaware/parsedmarc/commit/77132b3fc57358da6ea01fa9bb54034254c99c5e