ministryofjustice / analytical-platform

Analytical Platform • This repository is defined and managed in Terraform
https://docs.analytical-platform.service.justice.gov.uk
MIT License
8 stars 4 forks source link

📈 Document a Process for registering and sharing a Database/Table/Column from Ireland APDP to London APC using Lake Formation #4468

Open julialawrence opened 1 month ago

julialawrence commented 1 month ago

User Story

In order to keep the scope of the QuickSight MVP small, we have agreed to limit the initial implementation to a handful of pre-selected datasources. However, participants might want to add additional ones over the course of the testing, so to aid in that, we need a fully defined, documented process (manual or scripted) to share a resource from Ireland analytical-platform-data-production to London analytical-platform-compute

Value / Purpose

Documenting this would allow us to easily expand the scope of our MVP to new users and new datasets if required.

Useful Contacts

@julialawrence @michaeljcollins @jamesstott

User Types

Data Engineers

Hypothesis

If we implement this, we can be more flexible with the scope of our MVP testing, expanding it if it appropriate to do so.

Proposal

Document and test the process to do the following:

  1. Register a data location in anaytical-platform-data-production in Lake Formation in hybrid mode.
  2. Share resources from Ireland APDP into analytical-platform-compute Ireland.
  3. Share resources from Ireland APC to London APC
  4. Identity a data engineering user to be the admin of the shared datasource
  5. Assign them grantable permissions over the shared resource to allow them to continue managing access and grant it to others via the new UI.

Additional Information

https://docs.google.com/document/d/1Xwbvc8ipI2m6nlK3et-TUcrksjhV9MmNqEVFtVbtkOQ

Assume the sharing is done via named resources.

Definition of Done

Emterry commented 1 week ago

Document for testing sendong a lakeformation resource from AP Dev(ireland) -> AP Data Dev(ireland) then AP data dev(ireland) to AP data dev(London) https://docs.google.com/document/d/1bMk5onKLs1RC8oOXjLhlmAu84ZilAdxanvd7AAWqDRY/edit