search

awslabs / data-solutions-framework-on-aws

An open-source framework that simplifies implementation of data solutions.
https://awslabs.github.io/data-solutions-framework-on-aws/
Apache License 2.0
105 stars 14 forks source link

Data Lake Storage documentation on accessing buckets #621

Open PeterRayner opened 1 month ago

PeterRayner commented 1 month ago

Hi team

Im hoping we can add some documentation examples around accessing bronze/ silver/ gold buckets dynamically for CDK . for example for python:

storage = dsf.storage.DataLakeStorage(self, "MyDataLakeStorage") bronze_bucket_arn = storage.bronze_bucket.bucket_arn

In addition if there is a way to launch bronze /silver/gold in account A , but access logs bucket in audit account B, this would be great.

vgkowski commented 1 month ago

We can add a property in the DataLakeStorage to pass a custom access log bucket. In the cross scenario, the bucket policy on the access log bucket cannot be managed by the CDK stack so we also need to modify the AccesLogBucket to support granting the cross account logging from the DataLakeStorage