Data Mesh concepts support

[shachr]

Is your feature request related to a problem? Please describe. In recent years datamesh architectural style became very popular, and the ask is to organize data by domains, subdomains and bounded contexts ( some call it data domains ), additionally, be able to discovery data products rather than desperate dev assets such as pipelines, and data assets such as tables and topics, etc.

Describe the solution you'd like The solution should allow data stewards to create a taxonomy for their solution domain, optionally that taxonomy is governed by a central team. this will provide the framework for organizing data, such that users that discover can now also browse.

Next, all dev and data assets, such as topics, tables, pipelines, etc. should be group by a data product, e.g.: a data product that provides a realtime stream as a topic and historical data as table in the data lake, while pipeline could be the code portion of the data product where operational monitoring is observed. data product is also the unit that users discover when they search or browse the taxonomy.

Describe alternatives you've considered There are some alternatives, and it is implemented via many vendors, such as Alation, next data, and as far as i know azure support this through their Atlas offering, etc.

Additional context all modern architectural styles today are influenced by DDD, a data catalog should at the very least support these architectures, especially in the era of data mesh where DDD arrives to the analytical space.

[sureshms]

I am closing this issue in lieu of #12142. Please take a look at the proposal and provide feedback.