Plugin: How to handle disconnects?

[thenitai]

Hi there,

Love this project. I wrote a complete MongoDB plugin which parses the email (with attachments) and stores the email in MongoDB for further processing.

All great and dandy. However, I wonder what would be the best way to handle disconnects from MongoDB? Do I have to setup events within my plugin to catch when the plugin gets disconnected from MongoDB or is there something in your code?

Sorry, if this might be obvious, but the documentation does not show anything about this topic.

Thank you.

For those interested, the plugin is available at https://github.com/Helpmonks/haraka-queue-mongodb

[msimerson]

Thanks @thenitai , the plugin looks really good. I've just created a new wiki page with plugin recommendations

1. rename to haraka-plugin-mongodb. Although today your plugin only does queueing in MongoDB, tomorrow it might do more. There's already one mongodb plugin that stores connection metadata in MongoDB and I've heard interest from others wanting to save more complete connection metadata (think: copy the document assembly bits from my elasticsearch plugin and store in MongoDB instead). Extending plugins is easier than having an assortment of similar ones.

2. Add -mongodb to the services section of your .travis.yml (see plugin wiki page)

That will get a MongoDB instance fired up on Travis for you to test against. Use your favorite testing framework to write some basic tests, like config loading, connecting to MongoDB, and maybe even purposefully destroying the connection (by bouncing the mongodb server) to iron out how to get your code to handle reconnects. It looks like this SO issue has some useful information regarding how the mongdb client handles reconnections, and how you can influence it.

[thenitai]

@msimerson Awesome. I've renamed the repository now (I just didn't want to create a conflict. Though, other MongoDB plugins seem to be quite old now).

Actually, wanted to write some hooks for sending and store the results in MongoDB also. You think I should do this in the same plugin?

Will also use Travis stuff. Will get back here when done. Thank you.

[msimerson]

Actually, wanted to write some hooks for sending and store the results in MongoDB also.

You're in good company. :-)

You think I should do this in the same plugin?

Yes. Certainly.

What I'd also encourage is plan ahead for "feature options" in your plugin. A good way to handle this might be:

exports.register = function () {
    var plugin = this;
    plugin.load_example_ini();

    if (plugin.cfg.queue.enabled) {
        register.hook('data', 'enable_transaction_body_parse');
        register.hook('queue', 'queue_to_mongodb');
    }
    if (plugin.cfg.results.enabled) {
        register.hook('results', 'save_results_to_mongodb');
    }
}

exports.load_mongodb_ini = function () {
    var plugin = this;
    plugin.cfg = plugin.config.get('mongodb.ini', {
        booleans: [
            '-results.enabled',
            '-queue.enabled'
        ]
    },
    function () {
        plugin.load_mongodb_ini();
    }
}

Now your plugin does nothing by default, but flip a feature switch in the .ini file and it does queueing, flip two switches and it does results and queuing. Someone else might enable just results. It's very versatile and no hooks are registered and no code is run except when it is configured.

The other nice feature of not using the special hook_* function names is that your functions get better names that are indicative of what they do.

[msimerson]

Closing. Feel free to reopen if there's more to add.

[thenitai]

Ok, I've converted the plugin to a npm module (not on npm yet). I want to test this on our server, but for the live of me, I can't figure out where to put the module :(

If I use the plugin folder it tells me that there is no plugin called mongodb (I've named it haraka-plugin-mongodb).

Sorry, if this seems obvious.

[smfreegard]

I haven't created a plugin as a module yet, but for it to work properly - you'll need to do something like this:

cd /path/to/where/you/installed/haraka (e.g. /etc/haraka) npm install haraka-plugin-mongodb (use the git:// method to install it from your repo)

And Haraka should be able to find it then.

[thenitai]

Update:

Got it installed by going to the Haraka global installation path (I've installed Haraka globally) and dropped my code into the node_module folder.

Is there another location or is this how it should be?

[msimerson]

Hi @thenitai , please see Plugins.md. I've added an install section. Does that explain it well enough?

[thenitai]

@msimerson Thank you 👍

Another question, I'm running into a small issue where "connection" is not available within the shutdown hook. However, the doc says not to use "server" in an npm module. Though, I see people using "server" in their plugins. What is exactly working?

In general, it's a bit confusing jumping between docs and other plugins to figure out what exactly works. Is there a "official" source?

Thx

[msimerson]

where "connection" is not available within the shutdown hook

I would expect not.

However, the doc says not to use "server" in an npm module.

Which doc? (URL please)

[thenitai]

At https://haraka.github.io/manual/Plugins.html under section "Plugins as Modules":

Plugins loaded as modules are not compiled in the Haraka plugin sandbox, which blocks access to certain globals and provides a global server object. To access the server object, use connection.server instead.

I just saw that there is a haraka-results module. Should I store the connection in there?

[msimerson]

I think that paragraph is out of date. Try accessing the server object at run time in your plugin and I think you'll find that it works just fine.

there is a haraka-results module. Should I store the connection in there

Wouldn't do any good, as it only exists within the context of a connection (and said connections transactions).

[thenitai]

Yep, it works. Awesome. Ok, I'm going to put the MongoDB connection in server.notes.mongodb then. Thx

[thenitai]

I'm running into an issue with server.notes.

I'm storing the resulting connection to MongoDB now in server.notes.mongodb. However, when I try to access the database or even try to issue a close(), it tells me:

Cannot read property 'collection' of undefined

The funny thing is that when I store the very same database object in a global variable it works! I don't see any difference in how this is stored, but is there some "special" way I need to access the server.notes.variables?

This is the database object:

[NOTICE] [-] [mongodb] SERVER NOTES !!!{ mongodb:
   Db {
     domain: null,
     _events: {},
     _eventsCount: 0,
     _maxListeners: undefined,
     s:
      { databaseName: 'helpmonks',
        dbCache: {},
        children: [],
        topology: [Object],
        options: [Object],
        logger: [Object],
        bson: BSON {},
        authSource: undefined,
        readPreference: [Object],
        bufferMaxEntries: -1,
        parentDb: null,
        pkFactory: undefined,
        nativeParser: undefined,
        promiseLibrary: [Function: Promise],
        noListener: false,
        readConcern: undefined },
     serverConfig: [Getter],
     bufferMaxEntries: [Getter],
     databaseName: [Getter] } }

[thenitai]

I got it solved in the meantime. Issue was that I was only executing a hook on init_master. I've now added a hook on init_child also.

[thenitai]

@msimerson Alright, the plugin is done. We are running this already on a production server with over 40,000 emails a day. On top of storing incoming emails, we also record all outgoing results.

Not sure if there is a official way to announce this, but it is on npm as well https://www.npmjs.com/package/haraka-plugin-mongodb

[msimerson]

We don't really have an official way, but I have created a Plugin Registry and I've already added your plugin to that list.

[baudehlo]

Feel free to email the mailing list.

On Mar 3, 2017, at 4:46 PM, Matt Simerson notifications@github.com wrote:

We don't really have an official way, but I have created a Plugin Registry and I've already added your plugin to that list.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.