rodneyosodo
commented
9 months ago Proposed solution
Events from Things service, process them and store them in the internal DB
This approach is based on the idea that the Things service will publish events to the event store and the events processing service will subscribe to them. The service will process the events and store them in the internal DB. The service will expose REST API to query the history.
Pros
- This approach gives you complete control over how events are processed and stored. You can customize the storage format, handle complex event transformations, and ensure that the history records are stored in a way that aligns perfectly with your application's needs.
- By relying on events emitted by the Things service, you maintain a level of decoupling between different services. The Events/History service doesn't have to be tightly integrated with the Things service, and changes in the Things service can be handled independently.
Cons
- Repeating ourselves. We already have an event store that stores all events. Why do we need to store them in the internal DB?
- Implementing a custom event processing system can be complex and may require more development effort. You need to ensure that your event processing logic is robust, handles failures gracefully, and is scalable as your system grows.
- Depending on the frequency and volume of events, this approach may have performance implications. Processing events in real time and storing them in the internal DB can introduce latency, especially if the processing logic is resource-intensive.
DB triggers to automatically update history table(s)
This approach is based on the idea that we can use DB triggers to automatically update history table(s) when a record is inserted, updated, or deleted in the Things table.
Pros
- This approach is simple and requires minimal development effort. You don't need to implement a custom event processing system, and you can rely on the database to handle the event processing logic.
- Since the updates are triggered at the database level, there is a higher level of consistency.
- Eliminates the need for separate event processing infrastructure.
Cons
- This approach is tightly coupled with the Things service. If the Things service changes, you may need to update the triggers to ensure that the history records are updated correctly. Changes to the data schema might affect both services, potentially leading to a more tightly integrated system.
- Debugging issues related to database triggers can be challenging. The logic is executed at the database level, and debugging tools might be limited compared to application-level debugging.
- While automatic updates provide consistency, you might have limited control over the format and structure of the historical data.
- Triggers can add overhead to database operations, especially for frequent updates.
Unconsidered approach
Since we already have an event store that stores our events, there is no need to store them in the internal DB. We can use the event store to query the history.
For example, if I want to get the history of a thing, I can query the event store for all events related to that thing. The events endpoint will return a list of events, and I can use the different filters to get the events I need, e.g.: ?thingId=123, ?thingId=123&operation=update, ?thingId=123&channelId=456, etc.
Events can be easily sent to other services or systems for further analysis or action. For example, we can send events to a reporting engine to generate reports. We can send events to a dashboard application to display the history of a thing or channel. We can send events to a monitoring system to monitor the health of our system.
Conclusion
I would lean on using DB triggers to automatically update history table(s). This approach is simple and requires minimal development effort. You don't need to implement a custom event processing system, and you can rely on the database to handle the event processing logic. Since the updates are triggered at the database level, there is a higher level of consistency. Eliminates the need for separate event processing infrastructure. Based on this reference triggers can be used to audit data modifications. The downside of this approach is that it is tightly coupled to the service database which means that if the database schema changes, the triggers will need to be updated. Also, the changes/history will be stored in the same database as the service data which means that the database will grow faster and the performance will be affected. This negates the use of event-driven architecture. By definition, event-driven architecture is a software architecture pattern promoting the production, detection, consumption of, and reaction to events. An event can be defined as "a significant change in state". Event sourcing is an architectural pattern in which entities do not track their internal state utilizing direct serialization or object-relational mapping, but by reading and committing events to an event store, a type of database optimized for storage of events.
Therefore, I would propose a different approach. We already have an event store that stores all events. We would use the event store to query the history. For example, if I want to get the history of a thing, I can query the event store for all events related to that thing. The events endpoint will return a list of events, and I can use the different filters to get the events I need, e.g.: ?thingId=123, ?thingId=123&operation=update, ?thingId=123&channelId=456, etc.
arvindh123
Events/History service will track Thing and Channel (and Users?) change/update history. There are two potential approaches and the goal of this issue is to explore the drawbacks and benefits of them. 1) Use events from Things service, process them, and store them in the internal DB. 2) Use DB triggers to automatically update history table(s).