dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
395 stars 221 forks source link

Implement relation filtering on get_catalog macro #964

Open mikealfare opened 9 months ago

mikealfare commented 9 months ago

resolves #900

Problem

When get_catalog runs, it needs to return all relations within a schema, hence it cannot be parallelized or scale. It also means that all relations will be returned, not just those managed by dbt.

Solution

Allow for a set of relations to be passed in to limit the query against the database.

Checklist