search

apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://datastrato.ai/docs/
Apache License 2.0
677 stars 208 forks source link

[Improvement] Add new event bus listener for audit #3973

Open hanwxx opened 1 week ago

hanwxx commented 1 week ago

What would you like to be improved?

Based on the metadata change message, in our company has following consumption scenarios:

  1. Data governance: based on job lineage, metadata change messages will be sent to affected job owners, so they can decide whether to adjust jobs
  2. Metadata search: After processing, the medata data will be imported into ES to provide complex search capabilities.
  3. Audit scenario: the change operation is stored persistently for audit.

How should we improve?

So i want to add a audit listenr, listening to the change event, then send the message to the external MQ ( like RocketMQ） No response

FANNG1 commented 1 week ago

seems reasonable to add a event consumer to deliver events to message queues, Besides this, could event system satisfy your requirement?

hanwxx commented 4 days ago

Get it，event system can meet the requirement

jerryshao commented 3 days ago

It would be nice to have an audit log framework in Gravitino, like to define the format of audit log, the necessary information of each audit record. Besides, I think we can make the storage of audit log configurable, user can implement the specific storage like rocket mq, files, etc. For Gravitino, we don't have to bind to one storage.