Support additional metadata ingestion from Schema Registry for Kafka Connector

[nqvuong1998]

Is your feature request related to a problem? Please describe. We utilize the Kafka Connector within OpenMetadata, incorporating Avro schema definitions from Schema Registry for ingesting topics and their corresponding schema. However, OpenMetadata can ingest additional metadata from Schema Registry, encompassing details such as topic description, topic owner, column description, column tags, etc. Consequently, extracting specific metadata based on predefined rules proves highly beneficial.

Describe the solution you'd like

• Following rules: https://docs.confluent.io/cloud/current/sr/fundamentals/data-contracts.html#rules
• Additional metadata:
  • Topic description
  • Topic owners
  • Column description
  • Column tags
• Example Avro schema: "type": "record", "name": "TopicName", "doc": "TODO", "owners": ["David", "Jackson"] "fields": [ { "name": "fullName", "type": [ "null", "string" ], "doc": "TODO", "tags": ["pii"] }, { "name": "age", "type": [ "null", "int" ], "doc": "TODO", "tags": ["sensitive"] } ]

[harshach]

@ulixius9 lets check if we can do this for 1.5 otherwise move it to 1.6

[nqvuong1998]

Hi @harshach , Any update for this issue?