search

dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
297 stars 119 forks source link

Support for BigQuery federated queries (e.g. over CloudSQL, Spanner external connections) #141

Closed muscovitebob closed 7 months ago

muscovitebob commented 2 years ago

Describe the feature

It should be possible to query external connections, and define them as sources using this package.

This would use the EXTERNAL_QUERY mechanism instead of the currently used EXTERNAL TABLE mechanism, which currently only supports Bigtable and Cloud storage.

Describe alternatives you've considered

Currently we have plain models that incorporate the SELECT * FROM EXTERNAL_QUERY("cloudsql-instance", "SELECT * FROM database_name.external_table;") BigQuery syntax, but these have to live with the models and not with the sources, which these clearly are.

Additional context

This is specific to BigQuery.

Who will this benefit?

This feature will benefit every GCP user who leverages the external connection/federated query mechanism to access data stored in other systems hosted in GCP from their BigQuery data warehouse.

muscovitebob commented 2 years ago

The external connection these queries use should be considered infrastructure and is definitely out of scope for this feature, I don't think they should be managed in dbt at all.

github-actions[bot] commented 7 months ago

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

github-actions[bot] commented 7 months ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.