Reloading config with out restart or interruption

[DaAwesomeP]

I am going to make a system that replicates MaskMe. Users will login and be able to generate random email addresses that will either (with their option) forward to their real email address or be dropped. From looking at the documentation, I think that I can do this with an external database to manage whose email aliases are whose, logins, and etc. Unlike MaskMe, users will be able to send email from the aliases too. However, whenever a user logs in and adds an email alias, I don't want to have to restart the whole server. Then incoming emails could be missed. Is it possible to only reload the config file (in a split second) like Nginx does?

[baudehlo]

Config files are already reloaded automatically, though with a slight throttle after Haraka 2.6.1.

But it sounds like what you really want is a database. I would recommend using that instead.

On Apr 6, 2015, at 8:22 PM, P THE AWESOME notifications@github.com wrote:

I am going to make a system that replicates MaskMe. Users will login and be able to generate random email addresses that will either (with their option) forward to their real email address or be dropped. From looking at the documentation, I think that I can do this with an external database to manage whose email aliases are whose, logins, and etc. However, whenever a user logs in and adds an email alias, I don't want to have to restart the whole server. Then incoming emails could be missed. Is it possible to only reload the config file (in a split second) like Nginx does?

— Reply to this email directly or view it on GitHub.

[DaAwesomeP]

I can use a real database instead of the JSON config file? Does it integrate with MySQL or PostgreSQL easily?

[smfreegard]

Haraka doesn't use any database storage in it's core at all. That's the whole point - you can use anything you like in a plugin, the sky is the limit.

[DaAwesomeP]

I think I'll be able to use the JSON config file to config the server and then use my separate database and service to manage whose alias is whose and add/delete them.

How often (even with the throttle) is the config file updated? Is there a function to force its update?

[smfreegard]

The throttle that baudehlo was talking about above is a sedation timer to prevent repeated re-reading of the configuration files; it's 5 seconds, so you won't even notice it.

One thing to note though; if you're going to use JSON config files for this (and I don't really recommend it; if I were writing this, I'd probably use Redis), then you have to make sure that you write the configuration files like this:

1) Write output to myconfig.json.new 2) Perform some basic sanity checks to the file, e.g. make sure the JSON is parse-able, it's not < n entries and not > n entries 3) Rename from from myconfig.json.new to myconfig.json

Steps 1 and 3 are important as if you're writing a large file and try and write it directly to the filename that Haraka is watching for updates, then it might try and read the file before you've finished writing it. The rename ensure that the update is atomic and that Haraka always sees the completed file.

Writing a big JSON file periodically is going to be far less efficient than writing single entries to something like Redis though.

[DaAwesomeP]

I will be on a limited VPS, so too much persistent data in Redis is not really an option. If I don't use the JSON config file (and use MySQL or PostgreSQL and my own plugin), will I be able to simply use an add and remove function with Haraka or will I be carrying the same giant object in memory (instead of in the JSON file) in my own file as well as Haraka?

[smfreegard]

I don't really follow your meaning. If you use a big JSON file, then it's parsed and loaded into memory by Haraka. If you use Redis; that same data would instead be stored in Redis and queried by Haraka (and could be on a totally separate machine). If you use MySQL/PostgreSQL - you could read all of the data into Haraka at start-up and then have some sort of delta query to add/remove/update changed records, or you could query it 'live' as the mail comes in (which might not scale so well without a cache layer). It's really up to you.

[DaAwesomeP]

What methods would I use in a plugin to update the alias object?

[DaAwesomeP]

@smfreegard is this object accessible through a plugin?

[smfreegard]

What object??

[DaAwesomeP]

Let me rephrase that. No matter what method, how would I bypass Haraka's alias file and use my own plugin methods to supply it? I am effectively trying to make a database connector for it with a plugin. One way that Haraka could do this I assumed would be storing an object with the JSON contents of the alias file somewhere in memory. If Haraka uses such object (or some other method for accessing the alias data), then how could I manipulate it with my plugin?

[DaAwesomeP]

I found it. It does store an object with the JSON from the config file. So, how can I modify this behavior with a plugin? It will need to query my plugin each time and not simply load the object at startup.

[smfreegard]

That's exactly what it already does....

The aliases function runs for each recipient, therefore any changes will automatically be picked up. If there are not changes then the cached values will be returned - see https://github.com/baudehlo/Haraka/blob/master/docs/Config.md#reloadingcaching

[DaAwesomeP]

I understand, but how can I intercept this action with a plugin? Do I need to copy the current alias plugin into a new plugin and simply change the source so that it will query my system instead of the file or can I override the config file function some other way?

[baudehlo]

Config isn't hookable (mostly because it's blocking and we didn't want people to try and put Database access there), so yes you would have to copy it.

On May 22, 2015, at 10:27 PM, DaAwesomeP notifications@github.com wrote:

I understand, but how can I intercept this action with a plugin? Do I need to copy the current alias plugin into a new plugin and simply change the source so that it will query my system instead of the file or can I override the config file function some other way?

— Reply to this email directly or view it on GitHub.

[DaAwesomeP]

I guess I can watch the repo for changes to the alias plugin. Is there anything else that I need to worry about besides this file?

If I open source my database plugin then I will definitely stick a big warning at the front of the readme about blocking and slow downs. I will try to use as many asynchronous functions as I can to access to DB, but my particular server probably won't be receiving too much traffic, so I won't be able to accurately test any performance problems with large loads. I'll also try to keep it in mind while I make my plugin. I'll probably end up making some sort of caching system so that the database won't have to be queried constantly. I'll try to make it compatible with a few DBs to test performance and give options depending on the time I have to work on it.

[baudehlo]

Nothing more to watch than that file.

No need to add unnecessary caching. Don't try and run before you can walk :)

Also you can't do non-blocking DB access in node anyway, so you'll have to do it in a custom plugin, and load it every time.

On May 22, 2015, at 11:27 PM, DaAwesomeP notifications@github.com wrote:

I guess I can watch the repo for changes to the alias plugin. Is there anything else that I need to worry about besides this file?

If I open source my database plugin then I will definitely stick a big warning at the front of the readme about blocking and slow downs. I will try to use as many asynchronous functions as I can to access to DB, but my particular server probably won't be receiving too much traffic, so I won't be able to accurately test any performance problems with large loads. I'll also try to keep it in mind while I make my plugin. I'll probably end up making some sort of caching system so that the database won't have to be queried constantly. I'll try to make it compatible with a few DBs to test performance and give options depending on the time I have to work on it.

— Reply to this email directly or view it on GitHub.

[DaAwesomeP]

If you think that I can get away with the file, then I'll use it. It will save a lot of work. I'll still be making one plugin, and I'll still need a DB for the non-Haraka parts of my app, but this will be easier. Thanks a lot for both of your guidance.