search

MarquezProject / marquez

Collect, aggregate, and visualize a data ecosystem's metadata
https://marquezproject.ai
Apache License 2.0
1.78k stars 320 forks source link

Marquez dataset & job notifications' panel #1973

Open pawel-big-lebowski opened 2 years ago

pawel-big-lebowski commented 2 years ago

Several things can happen within a dataset or a job:

As a data engineer I can think of plenty of such scenarios. We do have in Marqeuz all the information to detect it but it is not that easy to visualize them. As a a data engineer, with a limited web UI skills, it is hard for me to make use of this information although I know how valuable it is.

In each dataset/job view we could have something like Notifications where such information can be presented. This would allow other Marquez contributors to implement their own notifications or alerts.

wslulciuc commented 2 years ago

@pawel-big-lebowski Love this idea! Added it to our roadmap. I'd be great to get a proposal written outlining how the web UI would display the notifications and any additional API changes needed to support aggregating the notification events in order to provide this level of transparency (similar to what you've outlined above). Would you be interested in writing the proposal?