Handler for message in IMAP queue failed with KeyError

[JamborJan]

Can you please help me understand this error? I am using Microsoft exchange online and thus dkim_domain='mydomainorg.onmicrosoft.com' appears. It seems there is an issue with that. I've setup DKIM as described by Microsoft here: Use DKIM to validate outbound email sent from your custom domain.

Have I done something wrong or is there improvement in the dmarc-metrics-exporter required?

Handler for message in IMAP queue failed.,
Traceback (most recent call last):,
  File "/home/dmarc-metrics/.local/lib/python3.7/site-packages/dmarc_metrics_exporter/imap_queue.py", line 61, in _consume,
  File "/home/dmarc-metrics/.local/lib/python3.7/site-packages/dmarc_metrics_exporter/app.py", line 94, in process_email,
    await handler(msg),
    metrics.update(event),
  File "/home/dmarc-metrics/.local/lib/python3.7/site-packages/dmarc_metrics_exporter/dmarc_metrics.py", line 57, in update,
    self.metrics[event.meta].update(event.count, event.result),
KeyError: Meta(reporter='google.com', from_domain='mydomain.org', dkim_domain='mydomainorg.onmicrosoft.com', spf_domain='mydomain.org')

[jgosmann]

Seems like a bug to me, but it is not clear to me what causes it. self.metrics is a defaultdict which should not raise KeyError, but add the key to the dictionary ... 🤔

[JamborJan]

Can I help you debugging this? Do you need an example report or other things?

[jgosmann]

An example report might help.

[jgosmann]

Should be fixed by PR #8.

[JamborJan]

I pulled in these changes and created a new docker image with them. Now I suddenly get an IMAP error again. Is it possible that there is a side effect causing this?

My fork I pulled your changes in is up to date here: JamborJan/dmarc-metrics-exporter The docker image can be found here: JamborJan/dmarc-metrics-exporter/pkgs/container/dmarc-metrics-exporter

INFO:     Waiting for application startup.,
INFO:     ASGI 'lifespan' protocol appears unsupported.,
INFO:     Application startup complete.,
INFO:     Uvicorn running on http://0.0.0.0:9797 (Press CTRL+C to quit),
INFO:     172.29.52.9:33652 - "GET /metrics HTTP/1.1" 200 OK,
...
INFO:     172.29.52.9:34718 - "GET /metrics HTTP/1.1" 200 OK,
Error during IMAP queue polling.,
Traceback (most recent call last):,
  File "/home/dmarc-metrics/.local/lib/python3.7/site-packages/dmarc_metrics_exporter/imap_queue.py", line 59, in _consume,
    async for uid, msg in client.fetch(1, msg_count):,
  File "/home/dmarc-metrics/.local/lib/python3.7/site-packages/dmarc_metrics_exporter/imap_queue.py", line 125, in fetch,
    match = mail_header_regex.match(line),
TypeError: cannot use a string pattern on a bytes-like object,
INFO:     172.29.52.9:34796 - "GET /metrics HTTP/1.1" 200 OK,

[jgosmann]

That is weird. The mail_header_regex that the error is complaining about is a bytes pattern.

[JamborJan]

First of all thanks for creating the docker image, I tested your image now and it works the same way mine did before thus I delete my fork. Unfortunately the TypeError: cannot use a string pattern on a bytes-like object, is still there. I attach an example.zip report I'm receiving from Google. I hope that helps to track down the problem.

[jgosmann]

That image is still using version 0.3 depending on aioimaplib 0.7.18. That version might not have a consistent return type for the IMAP calls. At least the changelog for version 0.9.0 hints on that.. The current development version of dmarc-metrics-exporter is already using the newer aioimaplib and hopefully fixes the problem. I want to test that version for a few days on my own server to make sure there is nothing major broken, but will probably make a 0.4.0 release later this week.

Building Docker images from unreleased versions seemed to complicated to me (at least to automate this). You'd have to have a script around the docker file building the Python wheel first and some logic to copy and install the file instead of downloading it from PyPI ...

[JamborJan]

Thanks for that feedback. Another week or two for stability reasons seems legit. I'm anyway very happy how responsive and willing to implement the whole Docker stuff you are. Thanks a lot for all of that. Great work!

[jgosmann]

You can now try with the jgosmann/dmarc-metrics-exporter:0.4.1 image. Note that there has been a breaking change in the configuration. The changelog describes the details.

[JamborJan]

This is working great and all is as expected. Thank you @jgosmann !