search

ministryofjustice / analytical-platform

Analytical Platform • This repository is defined and managed in Terraform
https://docs.analytical-platform.service.justice.gov.uk
MIT License
12 stars 4 forks source link

✨ Enable MWAA to read ECR in data-engineering-prod #5645

Open moj-supratikc opened 1 month ago

moj-supratikc commented 1 month ago

Describe the feature request.

See Slack thread here for details: https://mojdt.slack.com/archives/C06TFT94JTC/p1728513340561769

Describe the context.

Have added functionality to airflow to automatically keep images updated on pipelines without manual intervention, but airflow needs permission to read ECR from a different AWS account.

Value / Purpose

This will allow pipelines to keep image versions updated automatically without the need to manually change them.

This will be particularly useful in dev pipelines built from shared generic repos (e.g. airflow-create-a-pipeline) as our dev pipelines will always run on the latest image and flag any errors to be fixed before updating images in prod.

User Types

Data Engineers

simon-pope commented 1 month ago

To be presented at Request forum 14th October

simon-pope commented 1 month ago

To be presented in request forum 21st October

simon-pope commented 3 weeks ago

Request Forum: Related PR https://github.com/moj-analytical-services/airflow/pull/4509/files Take to refinement, may require a spike to determine how this may be done and the development cost.

julialawrence commented 3 weeks ago

We should follow the pattern laid down by MP for access. Relevant code links: https://github.com/ministryofjustice/modernisation-platform/blob/58c7fc263920cfbae45150546488815e8c57869e/terraform/environments/bootstrap/member-bootstrap/iam.tf#L413 https://github.com/ministryofjustice/modernisation-platform/blob/58c7fc263920cfbae45150546488815e8c57869e/terraform/environments/bootstrap/member-bootstrap/iam.tf#L462

Apply the above to the airflow role and that should enable the functionality.