Feature: Support for TLS-RPT

[ntimo]

Hello, I think it would be super amazing if the dmarc report parser could also parse the new TLS-RPT reports.

https://tools.ietf.org/html/rfc8460

[techsneeze]

Definitely interesting! I had not heard of this effort, but it is worth looking into. I wonder if it should be a separate script though. As is, this project would otherwise likely need to be renamed. They are also trying to report on different technologies and different authentication methods. I'd definitely be open to Pull requests if you have time to look into it.

[n0lz]

I would really appreciate support for that too. Is there already an idea on how/when this could become available?

[techsneeze]

I've since setup my own environment to get these reports as well. I don't have a way to process them yet, but my impression is that it has VERY small usage at this point. I definitely want to add it in the future, but it is low on my list of projects. If anyone wants to submit a pull request, I'd be happy to see it!

[n0lz]

But maybe you should delete usernames and passwords from the conf file :)

[freeze2404]

omg, thank you xD didn't think about this :-)

[freeze2404]

Just found some time to realize the TLS report parsing by myself: tls-report-parser.zip

It's totally based on your dmarcts-report-parser. I just changed several lines so that the compressed JSON data gets parsed instead of the XML data. Furthermore I changed the database structure, of course.

This script parses TLS reports only! I didn't enhance the original dmarcts-report-analyzer to be able to parse TLS reports additionally.

Please note: I only tested Google reports. The script only parses the first policy in the report. I didn't adopt the PHP files for displaying the data.

[jnew-gh]

I have begun to write the code to support TLS-RPT.

@freeze2404, thanks for submitting the json version of the parser. As you pointed out, except for the table structure, the procedure to parse json reports is identical to parsing xml reports; the program just has to be aware of which one it is dealing with. The only wrinkle with TLS-RPT is that the reports can also be submitted by HTTPS; that will take some more thought and will be integrated at some point in the future.

As a result, I think I am going to try to merge the TLS-RPT functionality into the current project rather than create a separate tlsts-report-parser. The same hopefully will apply to the viewer.

I am also going to use the same database to minimize the configuration and administration overhead. I've already managed to create the new tables (tls_report and tls_rptrecord) and populate tls_report with some records from gzipped files residing in a directory. The next step will be to add the report details to tls_rptrecord. After that, the next goal will be to parse the TLS reports attached to emails.

I'll try to keep everyone posted and hopefully submit a pull request soon.

[jnew-gh]

Progress!

I've written the code to populate the tls_rptrecord table with the failure_details data. Because emails can fail in different ways there can be multiple failure_details records in each report.

I've also managed to get the IMAP subroutine working. Now the script can parse TLS reports from an IMAP folder as well as zipped/gzipped reports residing in a file directory.

As well, the script can now differentiate between TLS and DMARC reports automatically. This means that both TLS and DMARC reports can reside in the same IMAP folder or file directory and the script will parse them and put them in the appropriate tables in the database.

Next up is to test parsing mbox, email and uncompressed files from a file directory.

[jnew-gh]

Success!

I have managed to get the parser handling both TLS and DMARC reports not only in IMAP but also in all of the report paths (zip/gzip, uncompressed, maildir and mbox).

Currently there is a lot of duplication in the new code which I am trying to refactor. Everything works but it's ugly. Hopefully soon I can submit a pull request.

I've also created an enhancement request for the viewer.

As I've also requested with the viewer, @techsneeze, could you please create a v2.0 branch for the parser?

[freeze2404]

Good to see this progress, thank you very much!

[calaad]

very interested in this feature !

[n0lz]

How is the status? Sadly there was no progress since several months.

[wioxjk]

Just checking in to see the status of this!

[alainwolf]

Unfortunately there is no related code to be found on @jnew-gh fork of the code.

[techsneeze]

Looks like @jnew-gh was waiting for me to make a new branch! Thanks for those that pinged this earlier today. I've created the v2.0 branch as suggested. Hopefully the code is around somewhere and can be submitted to keep this moving.

[jnew-gh]

Yay, @techsneeze made a v2.0 branch!

I'm pretty sure I have the TLS-RPT code in a local repository somewhere but it's been so long since I worked on it, I have no idea where I am in the code. I vaguely remember my local code was parsing TLS-RPT emails from both an IMAP source as well as from a compressed file (ah, and I see in my comment above that was the case).

Hopefully I can look at my code in the next week or so and upload something meaningful to the 2.0 branches of both the parser and viewer. If I don't, feel free to pester me again. :-)

[jnew-gh]

I've made a pull request to add TLS-RPT functionality in the v2.0 branch of the parser. I've also made a pull request to the viewer to take advantage of the new entries in the database.

This code should be considered alpha quality at the moment. I have done some basic testing and everything seems to work but I have not used it long enough to consider it usable in a production environment.

@ekalin, could you please review my PostgreSQL code if you are able. I have tried to duplicate your PostgreSQL additions as closely as possible but I don't have a PostgreSQL server to test on (I probably should install one).