📖 Implement terraform modules for sharing Athena resources via Lake Formation

[julialawrence]

User Story

As an AP Engineer, I would like to implement terraform for sharing named Athena resources across accounts and regions, in a similar way that the Observability Platform bootstraps new accounts.

Value / Purpose

The purpose of these module is to make adoption of lake formation for sharing resources simple for data producers. As for am immediate need, this will enable Lake Formation for managing resource access both for DPR and for QuickSight.

Useful Contacts

@julialawrence

User Types

No response

Hypothesis

If we create easily reusable modules for sharing Lake Formation resources, adopting it as an alternative to the current means of data sharing and access management will become simpler.

Proposal

Implement (a) terraform module(s) that do(es) the following:

• Registers a data location in Lake Formation
• Shares a database/table across accounts via named resource or tags approach
• Shares a database/table resource across regions via named resource or tags approach
• Designates an admin

Additional Information

Sharing resources with/from MP member account will also require tweaking deployment permissions for the github role used in MPE repository. Configuring the role with correct permissions is in scope for this story.

Definition of Done

• [x] Tags/Named resources approach agreed
• [x] Decision made regarding one or two modules
• [x] MPE deployment role permissions updated
• [x] Terraform module(s) implemented
• [ ] Sharing MP --> APDP and APDP --> MP tested
• [ ] Sharing London --> Ireland and Ireland --> London tested
• [ ] Final boss (Cross-acct, cross-region) tested
• [ ] Documentation updated
• [ ] Follow-on stories raised
• [ ] Functionality demoed at AP S&T

[Gary-H9]

We cannot add Localstack testing in the same manner that the observability module uses it. This is because the "Lake Formation is supported by LocalStack only in the pro image."

[Gary-H9]

Successfully ran through the Cross-Account Named Resource after receiving guidance. Then used this experience to review document - added/made headings uniform, added title, contents.

• Added screenshots where they were missing.
• Added details around adding a table
• Added specifics on populating S3 bucket, what the contents should be and the inclusion of a schema
• Added comments and moved sections where needed
• Added section on testing with Athena

[Gary-H9]

✅ - The branch lf-gary-test contains detailed description of successful cross-account, cross-region testing using the terraform-aws-analytical-platform-lakeformation module. This does not test sharing tables.

✅ - The branch lf-gary-test-2 contains detailed description of a WIP cross-account testing using the terraform-aws-analytical-platform-lakeformation module. This does not test sharing tables.

[Gary-H9]

Testing in both branches above now completed. Moving on to test with the inclusion of sharing tables.

[Gary-H9]

Closing post intensive testing in conjunction with the UI work today. local-exec or external functionality can be defined and added in later tickets if required.