De-register S3 location from Lake Formation when dataset is deleted

data-dot-all / dataall

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.

https://data-dot-all.github.io/dataall/

Apache License 2.0

226 stars 82 forks source link

De-register S3 location from Lake Formation when dataset is deleted #153

Closed dlpzx closed 1 year ago

dlpzx commented 1 year ago

Describe the bug

When we delete a dataset the created S3 location does not get deleted (no cleaned up)

How to Reproduce

Create dataset
delete dataset
check in Lake Formation

This makes the re-create/import of the dataset fail because the S3 bucket was already created/imported

Expected behavior

No response

Your project

No response

Screenshots

No response

OS

n/a

Python version

AWS data.all version

v1.1.1

Additional context

No response

dlpzx commented 1 year ago

This bug has been tested in the open-source version and it appears to be a customer particular issue. In the region where the customer deployed data.all datasets (Cape Town region), it was necessary to implement a custom resource to delete the Lake Formation storage location for the dataset. In contrast to the open-source version where we use the CloudFormation resource directly.

We can close this issue as solved.