Feature Request: Storing metadata of mails in a database and email itself to an archive.

[Bramzor]

Apparently there is no way to save metadata of a mail (sender/receiver/subject and information like bounced/delivered) to a database yet. There is a plugin for mongodb but doesn't have access to all this information.

Wouldn't it be nice if there would be a plugin for this so it is possible to use this information in a web UI or for other purposes? I guess such a general plugin should be an advantage for haraka compared to any other system.

[msimerson]

hey @Bramzor, take a look at the log.elasticsearch plugin (docs). It saves all the email metadata and a few of the message headers to Elasticsearch (as JSON docs). That plugin could be copied/extended to instead/also save the JSON doc to mongodb, CouchDB, or similar.

[Bramzor]

And saving the full email? Because meta data alone isn't enough in this case. Also the elasticsearch plugin is not complete, see this ticket: https://github.com/haraka/Haraka/issues/1455

[msimerson]

And saving the full email?

Correct, the ES plugin doesn't do it. And when you get around to doing it, you're going to have a lot more work than you're imagining to actually do so. If you store it, what format do you store it in? Raw? Decoded? With attachments inline or saved separately? Or just attachment metadata? There's a lot of issues lurking in there, and the answers to those questions can only be answered by identifying the use cases.

Also the elasticsearch plugin is not complete

That depends on your definition of complete. It's complete enough for me and my purposes. :-)

[Bramzor]

Me and someone else wants to use Haraka for specific usecases where we need to have the mail saved (yes, in raw format) so we can trigger a resend later on. I need it for R/C purposes and someone on the IRC channel for archiving purposes. So we will have to write this kind of plugin and any point in the right direction would be helpful. Or if anyone is open to write such a plugin for us (payed), we are always open for suggestions.

[msimerson]

In past cases (compliance) where I needed mail messages archived I used qmail-tap or postfix's always_bcc option. You could achieve a similar effect with Haraka by calling two queue plugins (whatever you already use plus something like test _queue that saves the messages off to wherever).

In any case, you will need to have a copy saved in raw format (for compliance, for re-indexing (after you discover an indexing bug, etc...)). Then, to enable additional fun features like full-text search, header searches, tracking mutating virus attachments, and other fun stuff, you'll want a copy of the message decoded from all the many email encodings to UTF-8 (for text) and binary+metadata for attachments.

[Bramzor]

If it would be possible to run 2 queue plugins than yes, that would already be a big part of the solution. But tried this and apparently, if you run both test_queue and queue_outbound, he only uses the test_queue. But that is good information, if such a thing would be possible...

For our purpose, raw format is sufficient as we only have to send it again afterwards and in that case, raw format is the most easy format. Than another point would be, how can we send that raw format back to the haraka server or is it better to use a different mail tool for this?

[msimerson]

how can we send that raw format back to the haraka server

Why not use exactly the format already being used when Haraka queues an email message? That's basically what you're doing, except not trying to deliver the message again until a signal you specify. Maybe you can just set the "try delivering again" timestamp to 100 years in the future?

[Bramzor]

The format used for queueing (test_queue) for example, is raw format and the queue directory also but with an additional json line at the top right? That is indeed a good format if it includes everything (including attachments for example). Looking at the test_queue code but not able to figure out why you cant run 2 queue plugins. What I could do is try to combine test_queue in the queue_outbound plugin.

[baudehlo]

@msimerson you probably want to read the IRC logs on this. The issue should remain closed. The author just needs to do the work to implement what he wants.

On Oct 2, 2016, at 5:38 PM, Bramzor notifications@github.com wrote:

The format used for queueing (test_queue) for example, is raw format and the queue directory also but with an additional json line at the top right? That is indeed a good format if it includes everything (including attachments for example). Looking at the test_queue code but not able to figure out why you cant run 2 queue plugins. What I could do is try to combine test_queue in the queue_outbound plugin.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.