Fetch issue - Githubissues

fl42 commented 2 years ago

Hello,

I still have an issue probably linked to Office365's IMAP implementation with version 0.4.2 I'll investigate asap

Any idea? I tried marking emails as unread without success.

INFO:     Waiting for application startup.                                                                                                                                                                       │
│ INFO:     ASGI 'lifespan' protocol appears unsupported.                                                                                                                                                          │
│ INFO:     Application startup complete.                                                                                                                                                                          │
│ INFO:     Uvicorn running on http://0.0.0.0:9797 (Press CTRL+C to quit)                                                                                                                                          │
│ Error during IMAP queue polling.                                                                                                                                                                                 │
│ Traceback (most recent call last):                                                                                                                                                                               │
│   File "/venv/lib/python3.7/site-packages/dmarc_metrics_exporter/imap_queue.py", line 59, in _consume                                                                                                            │
│     async for uid, msg in client.fetch(1, msg_count):                                                                                                                                                            │
│   File "/venv/lib/python3.7/site-packages/dmarc_metrics_exporter/imap_queue.py", line 132, in fetch                                                                                                              │
│     f"Expected group termination with ')', but got '{terminator}'."                                                                                                                                              │
│ dmarc_metrics_exporter.imap_queue.ImapClientError: Expected group termination with ')', but got 'b' FLAGS (\\Seen))''.

Thanks for your project!

jgosmann commented 2 years ago

I didn't implement the parsing of the server's response in a sufficiently general way. I'm working on it.

jgosmann commented 2 years ago

I pushed a commit (https://github.com/jgosmann/dmarc-metrics-exporter/commit/b0a14edf53306006f65e333907ce5e7d2f871f54) to the main branch that hopefully fixes this issue. Can you try it out and verify that the fix works?

fl42 commented 2 years ago

Thanks for the commit.

WIth b0a14ed, I got:

Error during IMAP queue polling.
Traceback (most recent call last):
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_queue.py", line 61, in _consume
    async for uid, msg in client.fetch(1, msg_count):
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_queue.py", line 124, in fetch
    uid = next(value for key, value in parsed_line[2] if key == "UID")
  File "/home/user/.local/lib/python3.9/site-packages/pyparsing/results.py", line 193, in __getitem__
    return self._toklist[i]
IndexError: list index out of range

jgosmann commented 2 years ago

I need to know more precisely what the response of the IMAP server looks like. I pushed a commit to main that allows to modify the logging configuration. Can you use that version and add the following to you configuration file?

    "logging": {
        "version": 1,
        "loggers": {
            "aioimaplib": {
                "level": "DEBUG"
            },
            "uvicorn": {
                "level": "INFO"
            }
        }
    }

⚠️ Please be aware that this might log the IMAP password in clear text!

The post the relevant lines from the log here. The relevant lines should start with these lines:

DEBUG:aioimaplib.aioimaplib:state -> SELECTED
DEBUG:aioimaplib.aioimaplib:Sending : b'XXXXX FETCH X:X (UID RFC822)\r\n'

where the X will can change (i.e. they won't match exactly). The relevant lines should end with either with a line like the following or when the program raises the exception you posted above:

DEBUG:aioimaplib.aioimaplib:tagged status b'XXXXX OK FETCH completed.'

Full sample output with test data:

DEBUG:aioimaplib.aioimaplib:state -> SELECTED
DEBUG:aioimaplib.aioimaplib:Sending : b'HGKD6 FETCH 1:1 (UID RFC822)\r\n'
DEBUG:aioimaplib.aioimaplib:Received : b'* 1 FETCH (FLAGS (\\Seen) UID 29 RFC822 {2618}\r\nReturn-Path: <noreply-dmarc-support@google.com>\r\nReceived: from 172.18.0.1 (HELO jans-mbp-870.speedport_w_921v_1_48_000); Fri Jan 07 16:52:21 UTC 2022\r\nContent-Type: multipart/mixed; boundary="===============8004666782478388790=="\r\nSubject: DMARC Aggregate Report\r\nFrom: noreply-dmarc-support@google.com\r\nTo: queue@localhost\r\n\r\n--===============8004666782478388790==\r\nContent-Type: message/rfc822\r\nContent-Transfer-Encoding: 8bit\r\nMIME-Version: 1.0\r\nContent-Disposition: attachment\r\n\r\nContent-Type: application/zip\r\nMIME-Version: 1.0\r\nContent-Transfer-Encoding: base64\r\nContent-Disposition: attachment;\r\n filename="reporter.com!localhost!1601510400!1601596799.zip"\r\n\r\nUEsDBBQAAAAAAIqOJ1RA51hJrAQAAKwEAAAwAAAAcmVwb3J0ZXIuY29tIWxvY2FsaG9zdCExNjAx\r\nNTEwNDAwITE2MDE1OTY3OTkueG1sPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgi\r\nID8+CjxmZWVkYmFjaz4KICA8cmVwb3J0X21ldGFkYXRhPgogICAgPG9yZ19uYW1lPmdvb2dsZS5j\r\nb208L29yZ19uYW1lPgogICAgPGVtYWlsPm5vcmVwbHktZG1hcmMtc3VwcG9ydEBnb29nbGUuY29t\r\nPC9lbWFpbD4KICAgIDxleHRyYV9jb250YWN0X2luZm8+aHR0cHM6Ly9zdXBwb3J0Lmdvb2dsZS5j\r\nb20vYS9hbnN3ZXIvMjQ2NjU4MDwvZXh0cmFfY29udGFjdF9pbmZvPgogICAgPHJlcG9ydF9pZD4x\r\nMjU5ODg2NjkxNTgxNzc0ODY2MTwvcmVwb3J0X2lkPgogICAgPGRhdGVfcmFuZ2U+CiAgICAgIDxi\r\nZWdpbj4xNjA3Mjk5MjAwPC9iZWdpbj4KICAgICAgPGVuZD4xNjA3Mzg1NTk5PC9lbmQ+CiAgICA8\r\nL2RhdGVfcmFuZ2U+CiAgPC9yZXBvcnRfbWV0YWRhdGE+CiAgPHBvbGljeV9wdWJsaXNoZWQ+CiAg\r\nICA8ZG9tYWluPm15ZG9tYWluLmRlPC9kb21haW4+CiAgICA8YWRraW0+cjwvYWRraW0+CiAgICA8\r\nYXNwZj5yPC9hc3BmPgogICAgPHA+bm9uZTwvcD4KICAgIDxzcD5ub25lPC9zcD4KICAgIDxwY3Q+\r\nMTAwPC9wY3Q+CiAgPC9wb2xpY3lfcHVibGlzaGVkPgogIDxyZWNvcmQ+CiAgICA8cm93PgogICAg\r\nICA8c291cmNlX2lwPmRlYWQ6YmVlZjoxOmFiYzo6PC9zb3VyY2VfaXA+CiAgICAgIDxjb3VudD4x\r\nPC9jb3VudD4KICAgICAgPHBvbGljeV9ldmFsdWF0ZWQ+CiAgICAgICAgPGRpc3Bvc2l0aW9uPm5v\r\nbmU8L2Rpc3Bvc2l0aW9uPgogICAgICAgIDxka2ltPnBhc3M8L2RraW0+CiAgICAgICAgPHNwZj5m\r\nYWlsPC9zcGY+CiAgICAgIDwvcG9saWN5X2V2YWx1YXRlZD4KICAgIDwvcm93PgogICAgPGlkZW50\r\naWZpZXJzPgogICAgICA8aGVhZGVyX2Zyb20+bXlkb21haW4uZGU8L2hlYWRlcl9mcm9tPgogICAg\r\nPC9pZGVudGlmaWVycz4KICAgIDxhdXRoX3Jlc3VsdHM+CiAgICAgIDxka2ltPgogICAgICAgIDxk\r\nb21haW4+bXlkb21haW4uZGU8L2RvbWFpbj4KICAgICAgICA8cmVzdWx0PnBhc3M8L3Jlc3VsdD4K\r\nICAgICAgICA8c2VsZWN0b3I+ZGVmYXVsdDwvc2VsZWN0b3I+CiAgICAgIDwvZGtpbT4KICAgICAg\r\nPHNwZj4KICAgICAgICA8ZG9tYWluPm15LXNwZi1kb21haW4uZGU8L2RvbWFpbj4KICAgICAgICA8\r\ncmVzdWx0PnBhc3M8L3Jlc3VsdD4KICAgICAgPC9zcGY+CiAgICA8L2F1dGhfcmVzdWx0cz4KICA8\r\nL3JlY29yZD4KPC9mZWVkYmFjaz5QSwECFAMUAAAAAACKjidUQOdYSawEAACsBAAAMAAAAAAAAAAA\r\nAAAAgAEAAAAAcmVwb3J0ZXIuY29tIWxvY2FsaG9zdCExNjAxNTEwNDAwITE2MDE1OTY3OTkueG1s\r\nUEsFBgAAAAABAAEAXgAAAPoEAAAAAA==\r\n\r\n--===============8004666782478388790==--\r\n)\r\n'
DEBUG:aioimaplib.aioimaplib:Received : b'HGKD6 OK FETCH completed.\r\n'
DEBUG:aioimaplib.aioimaplib:tagged status b'HGKD6 OK FETCH completed.'

fl42 commented 2 years ago

Hello,

With 47194d0,

Fetch sounds ok:

DEBUG:aioimaplib.aioimaplib:state -> SELECTED
DEBUG:aioimaplib.aioimaplib:Sending : b'NGNJ6 FETCH 1:1 (UID RFC822)\r\n'
DEBUG:aioimaplib.aioimaplib:Received : b'* 1 FETCH (UID 1771 RFC822 {9958}\r\nMESSAGE_SOURCE\r\n FLAGS (\\Seen))\r\n'
DEBUG:aioimaplib.aioimaplib:Received : b'NGNJ6 OK FETCH completed.\r\n'
DEBUG:aioimaplib.aioimaplib:tagged status b'NGNJ6 OK FETCH completed.'

I replaced message content by MESSAGE_SOURCE for confidentiality.

Output without DEBUG mode:

INFO:uvicorn.error:Waiting for application startup.
INFO:uvicorn.error:ASGI 'lifespan' protocol appears unsupported.
INFO:uvicorn.error:Application startup complete.
INFO:uvicorn.error:Uvicorn running on http://0.0.0.0:9797 (Press CTRL+C to quit)
INFO:uvicorn.access:127.0.0.1:42724 - "GET /metrics HTTP/1.1" 200
INFO:uvicorn.access:127.0.0.1:42726 - "GET /metrics HTTP/1.1" 200

So everything looks fine (not exception is raised).

However prometheus time series are not created:

$ curl -s http://127.0.0.1:9797/metrics | grep dmarc
# HELP dmarc_total Total number of reported messages.
# TYPE dmarc_total counter
# HELP dmarc_compliant_total Total number of DMARC compliant messages.
# TYPE dmarc_compliant_total counter
# HELP dmarc_quarantine_total Total number of quarantined messages.
# TYPE dmarc_quarantine_total counter
# HELP dmarc_reject_total Total number of rejected messages.
# TYPE dmarc_reject_total counter
# HELP dmarc_spf_aligned_total Total number of SPF algined messages.
# TYPE dmarc_spf_aligned_total counter
# HELP dmarc_spf_pass_total Total number of messages with raw SPF pass.
# TYPE dmarc_spf_pass_total counter
# HELP dmarc_dkim_aligned_total Total number of DKIM algined messages.
# TYPE dmarc_dkim_aligned_total counter
# HELP dmarc_dkim_pass_total Total number of messages with raw DKIM pass.
# TYPE dmarc_dkim_pass_total counter

I waited few minutes without much success (poll_interval_seconds = 10 in config)

Thanks for your support

jgosmann commented 2 years ago

That still leaves unclear what caused the previous failure. :(

It seems that the email got processed, but for some reason the DMARC report data was not extracted. Possible reasons for that:

Failure to extract the report from the email. But this should log a warning (Failed to extract report from email ...).
The report has been processed before. The reports's organization name and report ID will be stored in seen-reports.db under the "storage path" for deduplication_max_seconds (7 days by default). You could try either removing seen-reports.db or setting deduplication_max_seconds to 0 and then reprocess the report (just move the email from Archive/Aggregate back to the inbox).
The DMARC report does not contain any records. Should be easily checked by looking at the report XML. It should contain some <record> tags.

fl42 commented 2 years ago

I checked manually the DMARC report from Google, it has few records (xml in ZIP). I tried deleting the storage_path, so not an issue on this side.

Actually this is still the same issue:

ERROR:dmarc_metrics_exporter.imap_queue:Error during IMAP queue polling.
Traceback (most recent call last):
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_queue.py", line 61, in _consume
    async for uid, msg in client.fetch(1, msg_count):
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_queue.py", line 124, in fetch
    uid = next(value for key, value in parsed_line[2] if key == "UID")
  File "/home/user/.local/lib/python3.9/site-packages/pyparsing/results.py", line 193, in __getitem__
    return self._toklist[i]
IndexError: list index out of range

I had to remove logging config from config file and I added logging.basicConfig(level=logging.DEBUG) for the message to appear.

Idea: would you mind adding a --debug argument that sets root logger to level DEBUG, that would allow to show all debug messages more easily.

I'll keep you informed

EDIT: parsed_line is empty, which is not expected Adding DEBUG messages to the IMAP parser may help to troubleshoot

EDIT2: Why not using built-in parser email.message_from_bytes?

jgosmann commented 2 years ago

Idea: would you mind adding a --debug argument that sets root logger to level DEBUG, that would allow to show all debug messages more easily.

I'll consider it.

EDIT: parsed_line is empty, which is not expected

Yes, but it is not clear to my why that happens. I think this would require that the parsing of the IMAP response fails, but the output you posted looks fine to me. At least as long as {9958} matches the actual MESSAGE_SOURCE length ... (but why wouldn't it?)

I would like to enforce that the full response must be parsed, but the FETCH completed. is included in it which isn't standardized in the RFC. The tag with the result NGNJ6 OK is, but that's already removed by aiomaplib ...

Adding DEBUG messages to the IMAP parser may help to troubleshoot

Yes, changing this line to

fetch_response = Group(fetch_response_line[...]).set_debug(True)

should do the trick for now. Not sure how useful the output actually is, but maybe worth a try.

EDIT2: Why not using built-in parser email.message_from_bytes?

This is what I am essentially using to parse the email itself. However, the problem is one step before that: parsing the IMAP response containing the email. Unfortunately, IMAP has some annoying complexities and the aiomaplib is still fairly low-level ...

fl42 commented 2 years ago

Unfortunately I cannot provide a full example as it contains sensitive data. However I'll have a look of the issue.

Some time ago I wrote a small script to copy emails between IMAP servers. I didn't have any problems retrieving the messages, it may be useful for you. Note: I used built-in imaplib, not aiomaplib.

jgosmann commented 2 years ago

Given that I already implemented parsing of the IMAP fetch response myself, I figured it isn't that much more complicated to implement the IMAP client for the required functionality myself. While this probably won't resolve this issue by itself, I hope it gives a bit better debugging possibility (among some other reasons). I also fixed the logging configuration and added a --debug flag as requested.

All of this is currently on imapclient branch. Could you try it out and report back the debug output (with sensitive information removed)?

jgosmann commented 2 years ago

Apparently I broke Python 3.7 compatibility on that branch. You'll have to use Python 3.8 or newer until I fix it.

fl42 commented 2 years ago

Hello,

I tried to find the root cause without much success unfortunately.

With imapclient branch (fc90f03d434c4cbab0419d6b62f6f28e7857693b):

$ python3 -m dmarc_metrics_exporter --configuration config.json --debug
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
INFO:uvicorn.error:Waiting for application startup.
DEBUG:dmarc_metrics_exporter.imap_queue:Polling IMAP ...
INFO:uvicorn.error:ASGI 'lifespan' protocol appears unsupported.
INFO:uvicorn.error:Application startup complete.
INFO:uvicorn.error:Uvicorn running on http://0.0.0.0:9797 (Press CTRL+C to quit)
DEBUG:dmarc_metrics_exporter.imap_client:IMAP response: (<ResponseType.Untagged: '*'>, b'OK The Microsoft Exchange IMAP4 service is ready. [TABPADIAUAAyADYANQBDAEEAMAAzADYANgAuAEcAQgBSAFAAMgA2ADUALgBQAFIATwBEAC4ATwBVAFQATABPAE8ASwAuAEMATwBNAA==]\r\n')
DEBUG:dmarc_metrics_exporter.imap_client:IMAP server ready.
DEBUG:dmarc_metrics_exporter.imap_client:IMAP response: (<ResponseType.Untagged: '*'>, b'CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+\r\n')
DEBUG:dmarc_metrics_exporter.imap_client:IMAP server reports capabilities: b'CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+\r\n'
DEBUG:dmarc_metrics_exporter.imap_client:IMAP response: (<ResponseType.Tagged: 'tagged'>, b'a0 OK CAPABILITY completed.\r\n')
DEBUG:dmarc_metrics_exporter.imap_client:IMAP command 'b'a0'' completed with 'b'OK' b'CAPABILITY completed.\r\n''
DEBUG:dmarc_metrics_exporter.imap_client:IMAP response: (<ResponseType.ContinueReq: '+'>, b'Ready for additional command text.\r\n')
DEBUG:dmarc_metrics_exporter.imap_client:IMAP response: (<ResponseType.Tagged: 'tagged'>, b'a1 BAD Command Argument Error. 11\r\n')
DEBUG:dmarc_metrics_exporter.imap_client:IMAP command 'b'a1'' completed with 'b'BAD' b'Command Argument Error. 11\r\n''
ERROR:dmarc_metrics_exporter.imap_queue:Error during IMAP queue polling.
Traceback (most recent call last):
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_queue.py", line 48, in _poll_imap
    async with ImapClient(self.connection, self.timeout_seconds) as client:
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_client.py", line 214, in __aenter__
    await self._login(self.connection.username, self.connection.password)
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_client.py", line 311, in _login
    await self._command("LOGIN", login_writer)
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_client.py", line 297, in _command
    raise ImapServerError(name, tag.state, tag.text)
dmarc_metrics_exporter.imap_client.ImapServerError: IMAP error: Command LOGIN returned b'BAD' with response data: b'Command Argument Error. 11\r\n'
DEBUG:dmarc_metrics_exporter.imap_queue:Going to sleep for 10 seconds until next poll.

Maybe the simplest would be to forward reports to another IMAP server :P

jgosmann commented 2 years ago

I think I was missing a space in the IMAP login command. 😅 (The test server didn't seem to care either way. 😞) Pushed a fix, can you try again?

fl42 commented 2 years ago

The ... represents the message source (headers + body), unfortunately I cannot publish it.

DEBUG:dmarc_metrics_exporter.imap_client:Ignored untagged IMAP response: b'18 FETCH (UID 1797 RFC822 {9373}\r\nReceived: from ...\r\n FLAGS (\\Seen))\r\n'

Would you be able to reproduce the issue with a free Microsoft account?

jgosmann commented 2 years ago

ok, I might have figured out where it's going wrong. I used pyparsing but it only supports strings, no bytes objects. So I decoded the bytes object, but that's a bad idea:

The bytes might not constitute a valid encoded string.
Multiple bytes might decode to a single character and then the length given in bytes ({9373}) doesn't match the number of characters.

Seems like I have to implement the parser myself. At least I couldn't find a suitable library with a quick search.

fl42 commented 2 years ago

FYI

DEBUG:dmarc_metrics_exporter.imap_client:IMAP response: (<ResponseType.UNTAGGED: '*'>, b'1 FETCH (UID 1777 RFC822 {9228}\r\n...\r\n FLAGS (\\Seen))\r\n')

The string \r\n...\r\n is 9230 characters long (i.e., len("\r\n...\r\n")), not sure if it's expected

fl42 commented 2 years ago

Seems like I have to implement the parser myself. At least I couldn't find a suitable library with a quick search.

I did not understand why not to use Python built-in imaplib? :sweat_smile: Fetching emails don't need to be parallelized, then async code does not seem to be relevant here (unless it's a goal by itself to learn or whatever)

jgosmann commented 2 years ago

The async part is indeed not really required and more to get some more experience with it. Using the built-in imaplib would probably mean using threading and then I have to either mix two asynchronous programming models or swap out the HTTP server to. But more importantly the built-in imaplib has the exact same flaw of not parsing the FETCH response as aioimaplib. While one can get by with a hack and certain assumptions to extract the message, it is fragile and what sort of got us to this point.

fl42 commented 2 years ago

Sorry, I meant that the email module (not imaplib) is able to parse, as I used here?

jgosmann commented 2 years ago

I might be mistaken, but the email module parses only the RFC822 part (not the whole IMAP response), doesn't it?

fl42 commented 2 years ago

Consider the code snippet below:

"""
Strongly inspired by
https://medium.com/@sdoshi579/to-read-emails-and-download-attachments-in-python-6d7d6b60269
https://gist.github.com/johnpaulhayes/106c3e40dc04b6a6b516
"""

import json
import imaplib
import email

with open("config.json", "rt") as fd:  # dmarc-metrics-exporter config file format
    config = json.load(fd)

imap = imaplib.IMAP4_SSL(
    config["imap"]["host"],
    config["imap"]["port"]
)

typ, data = imap.login(config["imap"]["username"], config["imap"]["password"])
assert typ == "OK"

typ, data = imap.select("INBOX")
assert typ == "OK"

typ, data = imap.search(None, 'ALL')
assert typ == "OK"

msg_ids = data[0].split()
# print(msg_ids)

assert msg_ids, "No email in INBOX"

# Take the first email for this PoC
typ, data = imap.fetch(msg_ids[0], '(RFC822)')
assert typ == "OK"

msg = email.message_from_bytes(data[0][1])
# print(msg)

for part in msg.walk():
    if part.get_content_maintype() == 'multipart':
        continue
    if part.get('Content-Disposition') is None:
        continue

    with open("archive", "wb") as fd:
        fd.write(part.get_payload(decode=True))

$ file archive 
archive: Zip archive data, at least v1.0 to extract

$ unzip archive
Archive:  archive
  inflating: xxx.xml

You have to care whether it's a ZIP or a tar archive, but the code for this part should already exist...

Well done :smiley:

jgosmann commented 2 years ago

tl;dr: I'm still aiming to write a parser (the only way to cover all standard-conform cases), but will implement a quick fix similar to your proposal to be used in the meantime.

This code snippet

typ, data = imap.fetch(msg_ids[0], '(RFC822)')
assert typ == "OK"

msg = email.message_from_bytes(data[0][1])

relies on the IMAP server sending the RFC822 encoded message as literal string which is of the form {length}\r\n<string>. The imaplib (or aiomaplib) just splits the response at \r\n and thus it is accessible as data[0][1]. However, IMAP also allows for quoted strings of the form "<String>". In this case above code would no longer work because instead of [b'1 (RFC822 {length}', b'<String>', b')'] one would get [b'1 (RFC822 "<String>")']. I admit I would be surprised if a server would actually send a quoted string because literal strings are much more convenient to implement (e.g. no escaping or disallowed characters to deal with), but in theory it could happen. Thus, an correct, robust implementation cannot avoid proper parsing. I think that this should have been done by imaplib (or aiomaplib) to provide a proper abstraction, but it seems they offloaded the hard to the user of the library (the formal grammar for the FETCH response is actually a large part of the complete IMAP formal grammar).

In addition, I do not want to use the normal IMAP message numbering because it will change when the messages are moved to different mailboxes after processing. Thus, I actually fetch (UID RFC822) and in addition to the message, I have to parse the UID from the response. It adds a little bit more complexity because I cannot just do something like data[0][0] te get it. I suppose the server might even change the order in which the fields are returned, so I probably can't even count for the UID to be in data[0][0].

And a final bit of added complexity: The server may send additional fields in the response (usually FLAGS) which can go either in front or at the end (or in theory probably also in the middle). If an additional field goes in front comes with a literal string argument, the email itself suddenly is no longer in data[0][1], but maybe data[0][2] or data[0][3].

Because I value correctness and I'm already halfway there now (and some other reasons), I will still go in the direction of implementing a proper parser. However, most of the described problems are somewhat theoretic and in most cases the data[0][1] hack will work. So I'm also going to be a pit pragmatic and will probably implement a quick fix to make it work with the data[0][1] "hack" in the meantime.

jgosmann commented 2 years ago

I just released version 0.4.3 which hopefully fixes your problem in the meantime. I also made a different ~fix~ hack on the imapclient branch which should also fix the problem in a different way.

fl42 commented 2 years ago

Both v0.4.3 and a826e1354310d135105b6ed94059c38529f77b1c fix the issue :tada: Thanks for the hard work. I'm available to test new versions with Office365 if needed.

jgosmann commented 2 years ago

I'm using now my own parser (jgosmann/bite-parser) on the imapclient branch. That should fix the issue in a proper way. Hopefully without introducing new problems. If you like, you can give it a try before I package an official release.

fl42 commented 2 years ago

I installed bite-parser using pip install . (installation looks ok, importing module from Python prompt works)

Using 6f132d446da11dc34d5f955d27bd11f7ec180d4c, I got the following exception, maybe unrelated?

$ python3 -m dmarc_metrics_exporter --configuration config.json 
Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/__main__.py", line 3, in <module>
    from .app import main
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/app.py", line 17, in <module>
    from dmarc_metrics_exporter.imap_queue import ConnectionConfig, ImapQueue, QueueFolders
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_queue.py", line 12, in <module>
    from dmarc_metrics_exporter.imap_client import ConnectionConfig, ImapClient
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_client.py", line 23, in <module>
    from .imap_parser import response as response_grammar
  File "/home/user/dev/dmarc-metrics-exporter/dmarc_metrics_exporter/imap_parser.py", line 16, in <module>
    from pyparsing import dbl_quoted_string
ImportError: cannot import name 'dbl_quoted_string' from 'pyparsing' (/home/user/dev/dmarc-metrics-exporter/venv/lib64/python3.9/site-packages/pyparsing.py)

jgosmann commented 2 years ago

Oops, I overlooked an old pyparsing import (undetected by pylint because I assign the imported name to something else? And the tests were green because some development dependency uses pyparsing). Should be fixed now.

jgosmann / dmarc-metrics-exporter

Fetch issue #17