search

prashant280920 / killbill-kafka-consumer-plugin

Open source Contribution for Kill Bill to provide a plugin which can support usage push through kafka stream.
https://github.com/cashfree-oss/killbill-kafka-consumer-plugin
2 stars 0 forks source link

Daily Usage Records - Billable Events vs Daily Usage Totals #3

Open shaun-forgie opened 5 months ago

shaun-forgie commented 5 months ago

Kill Bill only records usage for a single day. Are you going to allow upstream services to generate billable events as messages where more than one event can occur on a single day? If so the consumer will need to consolidate / aggregate / totalise single events into a daily usage record.

To accomodate the different services generating messages it may be a good idea to have a number of different queues defined in the message broker to support both scenarios:

Upstream systems that have the ability to totalise and generate daily usage records can send messages to a different queue.

prashant280920 commented 5 months ago

A billable event refers to a usage or activity for which a charge is incurred. In this context, an upstream service may push usage data, whether aggregated or not, depending on its preferences. The osgikillBillAPI.recordRolledUpUsage()stores this usage data in the rolled_up_usage table. Previously, this usage data was pushed via an API call and could represent daily totals or multiple usage events within a single day. Similarly, services can now push usage events or aggregated daily usage to a Kafka topic instead of calling the API. The consumer then consumes, validates, processes, and stores each usage event in the rolled_up_usage table.

This usage data is later used during invoice generation to charge the merchant based on the usage pricing defined in their subscription plan.

Given that both aggregated daily usage and individual usage events need to be stored in the rolled_up_usage table, it's unclear why there is a need for two separate queues. Both types of usage data, whether aggregated or individual events, ultimately serve the same purpose of recording billable usage for invoicing.

shaun-forgie commented 4 months ago

Making a distincttion between aggregated and non aggregated usage records is important for upstream systems. I know that Kill Bill automatically consolidates all usage values it receives into a rolled up aggregated value. However when there is a requirement to process millions of usage messages a day it will be far more efficient to collect and consolidate usage values into daily totals outside of Kill Bill to reduce the load.

So this is not really about the Kafka plugin...but rather about message processing conventions that can be established to minimise the processing load on Kill Bill by allowing daily usage aggregations to be done outside of Kill Bill.