ministryofjustice / analytical-platform

Analytical Platform • This repository is defined and managed in Terraform
https://docs.analytical-platform.service.justice.gov.uk
MIT License
9 stars 4 forks source link

🎲 Provide GPU enabled Airflow containers #4755

Closed jacobwoffenden closed 2 weeks ago

jacobwoffenden commented 1 month ago

User Story

As an Analytical Platform engineer I want to provide base images for our Airflow service that container NVIDIA drivers So that DAGs can make use of the GPU compute capability we've enabled

Value / Purpose

Migration of Airflow to APC EKS has unlocked the ability to consume GPU enabled compute in a simpler fashion, however the base containers we provider do not include the NVIDIA drivers

Useful Contacts

@jacobwoffenden

User Types

Analytical Platform engineering

Hypothesis

If we add NVIDIA drivers to Airflow base images Then engineers can properly consume the GPU enabled compute

Proposal

Add NVIDIA drivers to Airflow base images

Additional Information

No response

Definition of Done

jacobwoffenden commented 1 month ago

Customer demand https://github.com/ministryofjustice/analytical-platform/issues/4907