datahub-project / datahub

The Metadata Platform for your Data and AI Stack
https://datahubproject.io
Apache License 2.0
9.93k stars 2.94k forks source link

kafka ingestion source ingests schemas for all topics and ignores topic_patterns #11907

Open guentherhackl-wgs opened 1 day ago

guentherhackl-wgs commented 1 day ago

Describe the bug When i load kafka topics and use topic_patterns it filters all the topics but correctly but still produces events for schemas for all topics on the cluster.

To Reproduce

  1. have a broker with 2 topics, both having a schema in confluent schema registry
  2. make a recipe that excludes one of the topics
  3. see all the schemas ingested anyway

Expected behavior the schemas schould also be filtered

Desktop (please complete the following information):