Netflix / metaflow

:rocket: Build and manage real-life ML, AI, and data science projects with ease!
https://metaflow.org
Apache License 2.0
7.8k stars 738 forks source link

Added support for private image repository credentials for Batch #1832

Open ArchieGertsman opened 1 month ago

ArchieGertsman commented 1 month ago

As mentioned in #1820, there is currently no way to authenticate with a private container image repo for Batch. In AWS, the ARN of a secret containing those credentials can be included in the job definition, which is what I've exposed from Metaflow. I added this as a Metaflow configuration called BATCH_CONTAINER_IMAGE_CREDS_SECRET, to the @batch decorator as repo_creds_secret, and to the batch cli as --repo-creds-secret.