Closed tinojantony closed 1 year ago
@tinojantony, can you try managing your data lake with AWS Lake Formation and enabling cross account access with Lake Formation?
Cross account access is working if you share tables using Lake Formation and resource link, this blog explain well how to perform this sharing: https://aws.amazon.com/blogs/big-data/controlling-data-lake-access-across-multiple-aws-accounts-using-aws-lake-formation/ The dbt-glue adapter is then able to read the data from your account database. The main limits is that it doesn't work cross region.
Describe the feature
We are planning to use DBT Glue which sources tables from other accounts in the organization. It seems dbt glue doesn't support/receive catalog id for cross account
Describe alternatives you've considered
As this feature is not supported, we can't use DBT glue. Instead we are using Glue code in PySpark itself.
Who will this benefit?
Glue jobs which requires cross account access for source data