🚀 Feature: Topic Processors

paulwer commented 1 day ago

🔖 Feature description

Novu topics could be extended with several new functionality to improve the targeted delivery audience by integrating custom processors into the @novu/framework package over the bridgeUrl with the user provided backend. Some of the provided concept-ideas may also be aplicable to introduce in the ui as well.

🎤 Why is this feature needed ?

Novu topics are by desing currently not secured, not strictly tied to a product database and sycing topics in advance is a huge task exspecialy for larger and existing systems, therefore this integration would help to fetch/validate data at runtime and only when its required. With such integration, users can also be more confident to use those topics, exspecialy for privacy related content and ensure the targeted users are the wanted ones. This would also reduce complexity when using a microservice architecture. when querying subscribers beforehand.

✌️ How do you aim to achieve this?

I will explain some concepts, which may or may not make sense :)

Integrate the user provided backend with processors like when using workflows. Several different processors, which can be tied to a topic name.

Processing Types:

beforeTrigger: this could be used to determine if additional users should be added to the topic
validateSubscribers: this could be used to validate if the subscriber is still valid or if he/she has to be removed from the topic, could also be used to execute it in batch, if the subscribers are very many
onAddSubscriber: could be used to do additional things, after a subscriber is removed
onRemoveSubscriber: could be used to do additional things, after a subscriber is removed

Topic Variables: Defining a controll/payload schema could help to pass some some additional properties to the processors. f.ex. to determine which subscribers to fetch.

Timeouts/TTL: Each Data returned by a processor should have a ttl, so in a production use-case the amount of executions is also limited/reduced.

Invalidation: With this aproach, existing processor data could be manualy invalidated to enforce a new run for the topic, when triggered next time. F.ex. someone is leaving an organization => invalidate all related topics.

Subscription Status: to overwrite some of the processors and remove the need to save additional data in the user-provided backend, subscriptions could have 3 status: 1. subscribed, 2. unsubscribed, 3. unknown => this would enable users to unsubscibe, even when the topicProcessor provides them => users could keep their custom preferences

Example:

topic(
    'organization',
{
   async beforeTrigger ({ payload }) => {
      // custom processors
      return false;
   },
   async validateSubscribers ({ payload, subscribers, isBatch }) => {
      // custom processors
      return { invalidSubscriberIds: [] };
   },
   async onAddSubscriber ({ payload, subscriber }) => {
      // custom processors
   },
   async onRemoveSubscriber ({ payload, subscriber }) => {
      // custom processors
   },
},
{
   payloadSchema: z.object({
      organizationId: z.string(),
      role: z.enum(['DEFAULT', 'ADMIN']).default('DEFAULT'),
   }),
   ttl: 6 * 60 * 60 * 1000,
});

🔄️ Additional Information

As discussed before via email, with @jainpawan21 this topic may already be on your timeline. I wanted to take the oportunity to share some of our current thoughts on this one, so maybe some others could share their opionions as well.

I would love to participate on discussions and maybe contribute at some points.

Thanks for your work and always for the support :)

👀 Have you spent some time to check if this feature request has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Contributing Guidelines

Are you willing to submit PR?

Yes I am willing to submit a PR!

linear[bot] commented 1 day ago

NV-4543 🚀 Feature: Topic Processors

ComBarnea commented 6 hours ago

@paulwer that's very interesting, would you mind arrange a session together and do a deep dive here? We have on the roadmap (Q1 2025) to do some more work on this, so together we might actually have the ability to merge this to the vision, as well as expedite this. I am available at tomer@novu.co. Let me know if you are open to a session.

novuhq / novu