HTTP API spec for external non-SMTP messenger integrations

[RaghavSood]

We are considering adding support for Firebase push notifications to Listmonk - would this be something that the upstream project would like to have sent up?

The proposed implementation is to add a Medium selector to campaigns which is one {Email | Firebase}, accept an array of firebase tokens per user, and support sending campaigns to lists in the same manner as the email system.

Since listmonk has a relatively easy interface for adding outgoing options, this simplifies things for us - just curious if the core listmonk project would be open to having us send these changes upstream

[knadh]

What I'd originally had in mind was using Go plugins to create Messenger plugins that can be dynamically loaded into listmonk. I've done this on another project. However, there are some issues with Go plugins that make them a bit of pain with a codebase that changes actively. The plugins have to be recompiled every time the host codebase changes. This may be impractical for a project like listmonk.

The other option is to bundle messengers into the core like you said, but that is not practical either because of the indefinite number of messengers (N number of SMS gateways for example) that can be written.

The third approach, I feel, maybe more practical and scalable. Define an HTTP messenger spec and build arbitrary messengers as external services. Maybe even a host HTTP server into which plugins are loaded. Or, instead of an HTTP spec, make the external host service an SMTP proxy. We've done this for some internal projects and works quite well.

[RaghavSood]

I agree that having some kind of plugin architecture, whether as a Go plugin or a interface compliant HTTP service would be a better option for integrating multiple options.

At the same time, Firebase is essentially the defacto way to do push notifications, unlike SMS or email, where you can have multiple options.

In any case, I'm happy to work on putting together the firebase option as a HTTP or plugin, if you're interested in working on specing out the rough interface I'd need to comply with. Perhaps we can open another RFC issue?

[knadh]

We can continue the thread here. Renamed the issue.

[RaghavSood]

I think the primary choice would be between a push and a pull approach.

Push Approach

Listmonk maintains control of the campaign, and makes batched HTTP POST requests to the plugins. This fits in will with the current email sending process, as it would let listmonk keep the batch selection logic and easily allow for pausing, progress monitoring, and graceful restarts from failures.

In this, we could define an outgoing message as consisting of a subject and a body. Listmonk can then make a post request along the lines of:

POST /send

{
    "subject": "Test subject",
    "body": "body text 123",
    "recipients": [
        {
            "uuid": "1-1-1-1-1",
            ... rest of the usee object
        }
     ]
}

The response to this would be a success object, with an optional failed field consisting of UUIDs that correspond to users for whom sending failed.

Here, each plugin could also define a GET /info endpoint which can return information like:

name
version
batch size
    different mediums might have different batch limits, for instance firebase only
    allows 100-500 push notifications to individual user tokens at a time I believe
{body, subject, total} length limit
    SMS has a character limit, so do readable notifications
required attributes
    plugins can define that a user must have an attribute present to be eligible for 
    use with a plugin, such as a phone number for SMS, or a firebase token for 
    firebase

Listmonk could then use the above to decide which users to include in a batch, and how big a batch should be, as well as using the name/version information to present available plugins in the campaign handling forms.

Pull Approach

Listmonk notifies a plugin when a campaign is ready. The plugin then uses the existing list and campaign APIs to pull the data and users, and sends it on it on.

This does feel a somewhat wrong solution, since progress handling, pausing, and error handling is considerably harder, since you would rely on the plugin to update you on each of the above, as opposed to listmonk being in control of the campaign state.

General Points

I think allowing plugins to specify a required field under user attributes for phone numbers, push tokens, webhook URLs, etc. is the correct way to approach it since it will not bloat the top level user object, and allows listmonk to trivially decide if a user compatible with a plugin by just checking for the existence of a key in the JSON attribs.

[knadh]

1. Couple minor tweaks to the API.

POST /send

{
    "subject": "Test subject",
    "body": "body text 123",
    "content_type": "richtext|html",
    "attachments": [
        "url": "https://site.com/123.zip",
        "name": "123.zip"
    ],
    "recipients": [subscriber{} ...]
}

2. Extent of integration I don't think there should be an /info call. The external messenger services should be completely agnostic and independent. listmonk should just do one-way message pushes like SMTP servers.

Just like how an smtp block is defined in the config, there can be an http block with similar configuration. There needn't be a concept of batches, again, like SMTP. HTTP max conns config can maintain a keep-alive pool.

3. Field semantics I think the concept of required fields will also be complicated. listmonk pulls subscribers from the DB in batches to process them, and attribs here here is merely a blob. Adding validation to that is going to have significant impact on the core campaign processing. The responsibility of ensuring that the required fields are present should be on the sender. Maybe maintain a separate list of users that have a Firebase token (an auto-segmention cron feature can further simplify this). If a messenger service receives a subscriber without required attributes, it can choose to return an error or silently ignore it.

4. Pull is needlessly complicated and impractical like you said.

[RaghavSood]

Minor tweaks

POST /send

{
    "subject": "Test subject",
    "body": "body text 123",
    "content_type": "richtext|html|plaintext",
    "attachments": [
        "url": "https://site.com/123.zip",
        "name": "123.zip"
    ],
    "recipients": [subscriber{} ...]
}

Plaintext is useful for SMS or other channels where rich formatting isn't an option.

Just like how an smtp block is defined in the config, there can be an http block with similar configuration. There needn't be a concept of batches, again, like SMTP. HTTP max conns config can maintain a keep-alive pool.

My concern there is that without some kind of batching operation with limits defined by the messenger, listmonk may send too many recipients at once - If a messenger has a hard limit on how quickly it can process things (such as firebase only allowing 100 recipients in a single invocation of the firebase API, or each SMS having to go out individually when using some SMS gateways), then listmonk sending a huge number of recipients may result in either listmonk timing out, or some form of silent failure on one side.

The responsibility of ensuring that the required fields are present should be on the sender. Maybe maintain a separate list of users that have a Firebase token (an auto-segmention cron feature can further simplify this). If a messenger service receives a subscriber without required attributes, it can choose to return an error or silently ignore it.

Agreed - this can be moved to a messenger responsibility fairly safely

[RaghavSood]

Actually, upon sleeping on it a little, I realized that we either need to do 1 HTTP request per outgoing message (so recipients is no longer an array), or provide an additional context field in the POST /send body that contains all the information required to render the templates.

Otherwise, having templating in the message won't work, especially for values that can't be found in the subscriber data (although given the campaign UUID and base URL of the listmonk install, they can probably be reconstructed)

[knadh]

Yep, one request per message is the simplest way forward. Recipients[] wasn't meant for batching subscribers, but for future proofing, just in case. Messenger.Push() already takes an array of e-mail addresses, for instance.

[RaghavSood]

Sounds reasonable - I'll do up a PoC based on this and see if it breaks

Thank you!

[knadh]

This is now complete and merged to master https://github.com/knadh/listmonk/commit/6cf43ea67493e771aa96c827b6ac3c37404138c4. Release shortly in v0.8.0.