📖 Update LakeFormation sharing module to automate manual onboarding steps

[julialawrence]

User Story

Currently, sharing data into another account requires the following manual steps:

• Opting in data locations, databases and tables into hybrid mode
• Granting principals LakeFormation data location permissions on registered path so that it doesn't break existing workflows.

This story is aimed at automating both of these steps.

Value / Purpose

Doing any process manually is both cumbersome and error-prone so automating it would allow the onboarding of data more easily. This is not only applicable to the QS MVP but also enables usecases such as Electronic Monitoring and Digital Prisons Reporting

Useful Contacts

@julialawrence

User Types

AP Ops

Hypothesis

If we automate the process, sharing data will become simpler for us and for our users.

Proposal

Implement the following: Using terraform external datasource and local-exec provisioner, implement

• Retrieval of all effective permissions both explicit and inherited for an s3 path
• Assigning data location permissions for the path to the principals
• Unknown: Figure out a way to maintain parity between existing AWS permissions and Lake Formation permissions

Using terraform null-resource and local-exec provisioner, implement

• Opt-in to hybrid for new location
• Using when=delete meta-argument, implement the removal of those permissions as well

Additional Information

This story should allow in this scope that an alternative solution is preferable, we don't want to get into a situation with everything looks like a nail with terraform

Definition of Done

• [ ] Approach agreed
• [ ] Proposal implemented
• [ ] Documented
• [ ] And tested
• [ ] And reviewed by another team member

[github-actions[bot]]

This issue is being marked as stale because it has been open for 60 days with no activity. Remove stale label or comment to keep the issue open.