ministryofjustice / analytical-platform

Analytical Platform • This repository is defined and managed in Terraform
https://docs.analytical-platform.service.justice.gov.uk
MIT License
12 stars 4 forks source link

🔱 Productionise Glue Copy Metadata script #6133

Open julialawrence opened 6 days ago

julialawrence commented 6 days ago

Context

Follows on from #5859

Currently, the python script which copies metadata across accounts is running locally. We need to decide and agree on where it's best to deploy it and conditions under which to run it.

Proposal

How do we run it?
Where we run it from
* CaDeT workflow
* Separate workflow
    *Linked only to relevant metadata (we can use the extracted metadata file)
* Take a stab at guessing timing and do it then. (lambda)
     * Means might not be current
* Other?

Spike requirements

Data Engineer 3 das

Definition of Done

tom-webber commented 3 days ago

https://github.com/moj-analytical-services/create-a-derived-table/pull/2465