I am missing the webhook processing

[Draccano]

I've just read the doc about DB sync across external API. The subscriber API works, but only one-directional - from outside server to listmonk.

The list monk does not have the logic that could notify the external server that subscriber credentials changed or he was deleted. And checking and comparing DB directly is not the smooth solution in my point of view.

@knadh would you implement a notify request system similar to messenger but for all CRUD operations?

[knadh]

Makes sense. Let me give this some thought, @Draccano.

[jugglingjsons]

Hey! has there been any movement on this, perhaps?

I am not sure if there is a public roadmap somewhere somewhere / are contributions on this front welcome. If they are and you have a proper idea on how you would like to see it implemented please let me know :)

[knadh]

Haven't started working on this, but had a vague idea on how this could be attempted.

• New admin tab to enable/disable webhooks and configure webhook endpoints.
• A goroutine listener that listens to an events channel.
• Every handler/action emits an event when the action is complete (this is going to be tricky, figuring out what to emit and sometimes how).
• The listener picks up the events and posts them to the webhook endpoints. Posting requires a fail-retry mechanism.

[dmitriy-komarov]

Hi @knadh

I think it could be nice to use experience of analytics services, like Amplitude. Here how to they catch events on their side:

• One HTTP(s) endpoint for all events ("type" is just a field of an event). So you can easiliy extend in future a list of envents Listmonk supports.
• JSON based protocol (both sides). Also easily extendable to add new event fields.
• Bulk events should be sent in one request to avoud thouthands of connections. Also due to all these events are very similar, such requests will have fine compression level.
• Maybe you just implement the protocol of Amplitude and let to specify API key of Amplitude? Listmonk users can start collecting statistics at the same moment they install Listmonk. If they need something special (own analytics), a field with custom endpoint URL can be implemented. In the code you can implement this as some Amplitude adapter for sending core JSON structures (later anyone can implement their own adapter if he really needs this).

[knadh]

One HTTP(s) endpoint for all events ("type" is just a field of an event). So you can easiliy extend in future a list of envents Listmonk supports.

Yep, standard webhook setup.

JSON based protocol (both sides). Also easily extendable to add new event fields.

There won't be no structures, but existing JSON request/response structures found in listmonk APIs.

Bulk events should be sent in one request to avoud thouthands of connections. Also due to all these events are very similar, such requests will have fine compression level.

It's not feasible (and is not ideal) to batch data structures. Standard HTTP keep-alive connections will solve this.

Maybe you just implement the protocol of Amplitude and let to specify API key of Amplitude?

Hardcoding a specific webhook service is not ideal. The webooks will be generic and if you want to use a service like Amplitude, or anything else, you can setup a proxy that does that.

[dmitriy-komarov]

Batch

I'm not sure that all servers will support keep-alive connections and provide a good compress level in such cases. Amplitude engineers already passed this step and finally decided to send multiple events at once. Collecting events in some buffer and sending them once a sec (5 sec / ... ) is pretty easy to implement IMO.

Hardcoding

As I wrote before, there is no need to hardcode, but you can give to Listmonk user a chance to create an adapter (as a plugin), and ship Amplitude adapter as a default adapter. Most of users do not want to host some proxy, deal with all this high-load stuff, or collect values manually, they just need to store all the events in existing analytics services where they can process the data as they wish.

Amplitude

Btw I'm not trying to do everyting as in Amplitude just because; just referencing to their experience. I also used their API to send analytics from my own app and implemented sending batch events. Also I just do not know is there any other alternative to Amplitude which has the same features, free plan and popularity. So I guess exactly such default adapter could be the most helpful for the biggest part of Listmonk audience. This is my vision, but the final decision is up to you, of course.

[knadh]

@dmitriy-komarov. You could always write an HTTP proxy between listmonk and other services that act accordingly. This intermediary service acts like an adapter or a plugin.

listmonk can only expose a univeral, standard interface. HTTP keep-alive connections, JSON bodies.

[jugglingjsons]

My three cents:

• don't batch webhooks - how will you process failures on specific messages then? To me it seems easier to handle them one by one.

In my opinion the system needs message retention - to pick up sending where it left of in case of bigger failures.

I would go with something similar to https://github.com/adjust/rmq - if you are willing to add some more 3rd party dependencies (redis and the package). It's an interesting challange to handle it on your own, but imho there is quite a bit of answers that have been given by the package already, like:

• dealing with retries
• dealing with 400/500 responses codes
• picking up where the process left off in case of failures.

What do you think ? I am game if project is open for contribution.

Also the way I see it - a minimal implementation is needed for now for what was requested. If someone needs a new webhook topic they can easily added by putting.

webhooks.push(&message{topic:Topics.EMAIL_SENT, ...body}});

where necessary

[knadh]

While I agree that a proper queue system would be robust, adding a dependency just for webhooks that only a tiny fraction of listmonk users will use is a no-go. Even if it's just a client lib that does a pub to something like an AMQP server, it complicates it. Are there examples of "consumer" web apps like listmonk that publish events via non-HTTP interfaces?

[yolossn]

I think this feature can be built without having to add a dependency on a queue system. Instead of handlers emitting actions I think it would be better to trigger webhook events based on the changes in the DB, If a generic layer like this is built we can create a generic Event trigger system similar to the one Hasura provides. This allows the user to setup which changes should trigger a webhook call.

[tcurdt]

@knadh in another project I also did not want to go crazy with dependencies just have some queuing functionality.

I found https://github.com/nsqio/go-diskqueue to work OK enough. And it's so small that you could even pull it into the code base.

Just my 2 cents.

[relikd]

Came here from for a webhook but would be happy with a messenger too. What I am missing is a webhook which is fired once per campaign (instead of each recipient). For non-email endpoints, e.g., archive, Telegram / Signal bot, etc.

[Mokkapps]

Would love to see webhooks to trigger actions if a campaign finished or a subscriber unsubscribed