[FEATURE] It is very important to support the centralized management of metadata

[Narcasserun]

As a stream computing platform, it is particularly important for streamx to support metadata management，I will give the design of metadata. Metadata can improve the development efficiency and let users only care about business logic. If it can support data governance at the same time, it will make streamx more powerful

1.Architecture diagram

2.Key design points

• SPI based metadata acquisition
• Metadata can be acquired in real time without external storage
• Plug in metadata, which can run independently

  3.Integration with streamX / Other platforms

• One key generation of streamx source table and sink table，Users only care about business logic
• Metadata provides restful services for other platforms to use directly

[Al-assad]

@Narcasserun Good idea. Looking forward to your design for data governance.

[Narcasserun]

Metadata is already under development. I divide metadata management into two parts: data source management and metadata. I came across a decision-making problem: does data source management need to be persisted to DB?

[Al-assad]

hi @Narcasserun, i don't think persistent storage of meta data is necessary. The cost of loading Kafka, MySQL, Hive meta info on demand is very low. However, there are additional data consistency issues that need to be addressed for meta data persistent storage.

[BruceWong96]

Hi @Narcasserun @Al-assad , i think persistent storage of meta data is necessary. Because, it is important for the data lineage and the metadata search.

[zfanswer]

Any update util Aug, 2023？