search

aws-samples / dbt-glue

This repository contains de dbt-glue adapter
Apache License 2.0
94 stars 60 forks source link

Cross account access for dbt sources #47

Closed tinojantony closed 1 year ago

tinojantony commented 2 years ago

Describe the feature

We are planning to use DBT Glue which sources tables from other accounts in the organization. It seems dbt glue doesn't support/receive catalog id for cross account

Describe alternatives you've considered

As this feature is not supported, we can't use DBT glue. Instead we are using Glue code in PySpark itself.

Who will this benefit?

Glue jobs which requires cross account access for source data

OElesin commented 2 years ago

@tinojantony, can you try managing your data lake with AWS Lake Formation and enabling cross account access with Lake Formation?

menuetb commented 1 year ago

Cross account access is working if you share tables using Lake Formation and resource link, this blog explain well how to perform this sharing: https://aws.amazon.com/blogs/big-data/controlling-data-lake-access-across-multiple-aws-accounts-using-aws-lake-formation/ The dbt-glue adapter is then able to read the data from your account database. The main limits is that it doesn't work cross region.