absmach / magistrala

Industrial IoT Messaging and Device Management Platform
https://www.abstractmachines.fr/magistrala.html
Apache License 2.0
2.48k stars 673 forks source link

Add Examples For Event Sourcing #2032

Closed rodneyosodo closed 5 months ago

rodneyosodo commented 1 year ago

Description

Add examples for events emitted by mainflux services when using nats JetStream and rabbitmq streams as the event source. This will help in documenting the events and also help in understanding how the events are emitted and how to view them.

Event Sourcing

Mainflux uses Event Sourcing. This means that some of the data is stored in the form of events. This allows mainflux to easily integrate with other systems and to have a clear audit trail of all the changes that happened in the system. With event sourcing, data changes are saved in domain events. These events are held in the chronological order of occurrence. These events can be replayed to get to the current state of the data. For example, when a user registers, an event is emitted to the event store. This event can be replayed to get the current state of the user. This is useful when you want to integrate with other systems. You can just replay the events to get the current state of the data.

Every function call in the things and users service emits an event. Some function calls in the mqtt service emit events but not all. Bootstrap service emits and consumes events from the things service. For example, assume you have connected a thing to a channel and have created a bootstrap configuration for that thing. When you delete the thing, the bootstrap configuration for that thing is also deleted. This is done by listening to the thing.deleted event in the bootstrap service. When this event is received, the bootstrap configuration for that thing is deleted.

These events are then stored in an event source which can either be redis streams, nats JetStream or rabbitmq messages. By default, the events are stored in Nats JetStream. With build tags, you can change the event source to either Redis streams or rabbitmq messages. The build tags are redis and rabbitmq respectively.

Nats JetStream

Using this PR as the reference, we can see how to use Nats JetStream as the event source.

  1. Build mainflux

    MF_ES_STORE_TYPE=nats make dockers
  2. Run mainflux

    MF_ES_STORE_TYPE=nats make run up args="-d"
  3. Simulate events using the e2e tool

    go run tools/e2e/cmd/main.go -n 1 -N 1

    This will create 1 user, 1 thing, 1 channel and 1 group. It will do CRUD operation on the entities and will also connect the thing to the channel. This will generate a lot of events together with mqtt events as it will publish messages to the channel.

  4. To simulate bootstrap events, you can use the UI. Do CRUD operations on the bootstrap configurations and you will see the events in the nats logs. During the update provide actual certificates to simulate the bootstrap.updated event.

  5. Once the events are generated, you can use the nats-cli to view the events.

    nats stream view
  6. Record the events to the documentation following this approach under the events page you can see the events that were generated. Do the same for Nats.

Redis Streams

This has already been implemented in master branch. You can use the same approach to document the events and make sure they are aligned. Use MF_ES_STORE_TYPE=reids when building mainflux and running it. You can use the redis-cli to view the events or redis insight.

RabbitMQ Messages

This has not been implemented yet. You can use the same approach to document the events. Use MF_ES_STORE_TYPE=rabbitmq when building mainflux and running it. You can use the rabbitmq-cli to view the events or rabbitmq management.

Requirements

Resources

rodneyosodo commented 5 months ago

@dborovcanin This can be closed as it has been solved https://docs.magistrala.abstractmachines.fr/events/