Open thx1111 opened 1 month ago
Can you provide a sample message file? Maybe in a Github gist or something?
Are each of the dmarc report files required to be manually extracted from the dmarc email, and moved to another directory, before processing?
No. I'm processing individual messages (in EML format) and the tool extracts the attached reports properly (usually).
Am I missing this utf8 package? Or the mime package?
You don't need to worry about these dependencies since the tool written in Go. All the package requirements are settled at build time, so you don't have to worry about these dependencies in your runtime environment.
It's interesting. I have no mime packages installed on my system and can process eml files as well. If you cannot provide sample file, you can post here output of
file <path_to_file>
head -n 2 <path_to_file>
But according to this https://cs.opensource.google/go/go/+/refs/tags/go1.22.5:src/mime/type_unix.go - it's possible that some packages are needed to detect mime type. I have "/usr/share/mime/globs2" file on my Fedora system and it belongs to shared-mime-info package
Here's a recent random sample dmarc file from Google with Content-Type: application/zip;
:
sampledmarcfile.txt
Here's another from Comcast with Content-Type: multipart/mixed;
and Content-Type: text/plain; charset="utf-8"
sampledmarcfileutf8.txt
Both files include the header MIME-Version: 1.0
.
I also have the "/usr/share/mime/globs2" file, generated by update-mime-database /usr/share/mime
from the Arch Linux "shared-mime-info 2.4-1" package.
It may or may not be important that these files are NOT specifically .eml
files, and instead, are Courier Maildir++
format files, delivered through postfix
. There may be some subtly important difference in the file formats. For instance running file
against some random .eml
file gives:
Birthday.eml: Unicode text, UTF-8 text, with very long lines (835), with CRLF line terminators
while running file
against the above sample files gives instead, for instance:
sampledmarcfileutf8.txt: SMTP mail, ASCII text
If you copy one of the files out of the Maildir and rename it as a .eml
file, does this tool process it?
Ha! Yes.
I moved the sample files to a directory "dmarctest" and changed the extensions to .eml
. Here is the output:
$ dmarc-report-converter
[INFO] files: found 2 eml file(s), extract attachments to /home/james/dmarctest/
[INFO] extractAttachment: found attachment: google.com!nurealm.net!1720828800!1720915199.zip
[INFO] extractAttachment: save attachment to: /home/james/dmarctest/google.com!nurealm.net!1720828800!1720915199.zip
[WARN] extractAttachment: inline MIME headers are not currently supported, skip
[INFO] extractAttachment: found attachment: comcast.net!nurealm.net!1710460800!1710547200.xml.gz
[INFO] extractAttachment: save attachment to: /home/james/dmarctest/comcast.net!nurealm.net!1710460800!1710547200.xml.gz
[INFO] files: found 2 input files in /home/james/dmarctest/
[INFO] ReadParseZIP: read file google.com!nurealm.net!1720828800!1720915199.xml from zip
[INFO] merge: 1 report(s), grouped by key 'comcast.net!dmarc-admin@alerts.comcast.net!nurealm.net'
[INFO] merge: 1 report(s), grouped by key 'google.com!noreply-dmarc-support@google.com!nurealm.net'
[INFO] output: write to file /var/lib/dmarc-report-converter/outputs/2024-03-14-nurealm.net/dmarc-admin@alerts.comcast.net-v2-1710561012-nurealm.net.html
[INFO] output: write to file /var/lib/dmarc-report-converter/outputs/2024-07-12-nurealm.net/noreply-dmarc-support@google.com-6171381054545036256.html
Of course, now I'm curious - what happened there?
For reference and for instance, the sample files have the original names "1720949687.V804I914868M641386.topaz:2,S" and "1710608371.V804I910597M983359.topaz:2,". These are typical Maildir file names. Dovecot has some explanation of the filename format at https://doc.dovecot.org/admin_manual/mailbox_formats/maildir/ .
As an additional check, I went back and renamed those sample files to "sampledmarcfile" and "sampledmarcfileutf8", without the explicit .eml
suffix - to see if the Maildir name format itself might have been the cause of the naming problem - and re-ran dmarc-report-converter
. But, same error messages:
$ dmarc-report-converter
[INFO] files: found 2 input files in /home/james/dmarctest/
[ERROR] files: mimetype text/plain; charset=utf-8 not supported in file /home/james/dmarctest/sampledmarcfile, skip
[ERROR] files: mimetype text/plain; charset=utf-8 not supported in file /home/james/dmarctest/sampledmarcfileutf8, skip
[ERROR] processFiles: reports list is empty
The naming problem, then, really does appear to be connected to the .eml
filename extension.
Thank you for debugging, it looks like a bug
I'd consider this an enhancement request instead of a bug, but it's your project. 😉
I ran into this issue but worked around it. I use AWS SES to receive DMARC reports which are placed in an S3 bucket with a hashed filename with no file extension. I have a script that downloads the files from S3 and renames them to have an eml file extension.
@moorereason While I can appreciate your high pain threshold, I'm inclined toward "bug", and not "enhancement". Still, we might agree that this .eml
suffix issue is not a "feature". Either way, thanks very much for your suggestion to test the file name extension! I certainly wouldn't have expected something like that.
I use AWS SES to receive DMARC reports which are placed in an S3 bucket with a hashed filename with no file extension
Interesting way. I use input.imap
to download and process reports directly from email server :)
The reason for this unexpected behavior is simple
Arch Linux go 2:1.22.4-1
These errors apply to every single dmarc report received, including from various different sources.
Are each of the dmarc report files required to be manually extracted from the dmarc email, and moved to another directory, before processing? Or, is this a bug?
https://pkg.go.dev/unicode/utf8 says:
https://pkg.go.dev/mime says:
Am I missing this
utf8
package? Or themime
package? Or something else?