search

open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
https://open-metadata.org
Apache License 2.0
5.21k stars 991 forks source link

table filter pattern: exclude empty tables #12842

Open DovileKr opened 1 year ago

DovileKr commented 1 year ago

Is your feature request related to a problem? Please describe. I am working on ingesting metadata from COTS solutions. Those solution have many pre-built tables for many scenarios. For example I have a retail system that covers POS, Inventory, Scheduling, Staff. However, we only use it for POS transactions.

Why empty tables don't add value for data search and discovery?

  1. This increases the amount of "dead-end" search results. Person searches for table or column. Gets a result; based on on schema looks promising. Goes to see the data (maybe profile, maybe sample data) and finds that it is empty. First thought - sample data / profile wasn't ingested. Then sees it was ingested, and realizes that table is simply empty. they have to restart the search. I would find it very annoying.
  2. misrepresents "data asset" count.

Describe the solution you'd like On metadata ingestion, have an option to exclude tables with zero rows (mainly for BIG databases like oracle, mssql). So that I don't have to remove one by one.

Describe alternatives you've considered Run table metrics, identify all empty tables and have a very long "exclude by table name" filter pattern.

Additional context Add any other context or screenshots about the feature request here.

DovileKr commented 7 months ago

hello, do we have any plans for this enhancement?