search

GoogleCloudPlatform / automlops

Build MLOps Pipelines in Minutes
Apache License 2.0
143 stars 34 forks source link
ci-cd google-cloud-platform jupyter-notebook machine-learning mlops pipeline vertex-ai

readme

AutoMLOps

AutoMLOps is a service that generates, provisions, deploys, and monitors CI/CD integrated MLOps pipelines, bridging the gap between Data Science and DevOps. AutoMLOps provides a repeatable process that dramatically reduces the time required to build MLOps pipelines. The service generates a containerized MLOps codebase, provides infrastructure-as-code to provision and maintain the underlying MLOps infra, provides deployment functionalities to trigger and run MLOps pipelines, monitoring capabilities to monitor models deployed to live endpoints, and optional automatic retraining of models on a recurring basis or if anomalies are detected during monitoring.

AutoMLOps gives flexibility over the tools and technologies used in the MLOps pipelines, allowing users to choose from a wide range of options for artifact repositories, build tools, provisioning tools, orchestration frameworks, and source code repositories. AutoMLOps can be configured to either use existing infra, or provision new infra, including source code repositories for versioning the generated MLOps codebase, build configs and triggers, artifact repositories for storing docker containers, storage buckets, service accounts, IAM permissions, APIs needed to run pipelines, RESTful services to allow for triggering and running pipelines asynchronous, and Cloud Scheduler jobs for triggering and running pipelines on a recurring basis.

These automatic integrations empower data scientists to take their experiments to production more quickly, allowing them to focus on what they do best: providing actionable insights through data.

Install

Install AutoMLOps from PyPI: pip install google-cloud-automlops

Or Install locally by cloning the repo and running pip install .

Dependencies

Usage

AutoMLOps provides 2 functions for defining MLOps pipelines:

AutoMLOps provides 6 functions for building and maintaining MLOps pipelines:

For a full user-guide, please view these slides.

List of Examples

Training

Inferencing

Supported Tools and Technologies

Artifact Repositories: Stores component docker containers

Deployment Frameworks: Builds component docker containers, compiles pipelines, and submits Pipeline Jobs

Orchestration Frameworks: Executes and orchestrates pipelines jobs

Submission Service Compute Environments: RESTful service for submitting pipeline jobs to the orchestrator (e.g. Vertex AI, Cloud Composer, etc.)

Provisioning Frameworks: Stands up necessary infra to run MLOps pipelines

Source Code Repositories: Repository for versioning generated MLOps code

Prerequisites

Generate

In order to use AutoMLOps.generate(...), the following are required:

Provision

In order to use AutoMLOps.provision(...) with provisioning_framework='gcloud', the following are recommended:

In order to use AutoMLOps.provision(...) with provisioning_framework='terraform', the following are recommended:

Deploy

In order to use AutoMLOps.deploy(...) with use_ci=False, the following are required:

In order to use AutoMLOps.deploy(...) with use_ci=True, the following are required:

Monitor

In order to use AutoMLOps.monitor(...), the following are required:

GCP Services

AutoMLOps makes use of the following products by default:

AutoMLOps will makes use of the following products based on user selected options:

  1. if artifact_repo_type='artifact-registry', AutoMLOps will use:

  2. if use_ci=False and orchestration_framework='kfp' and deployment_framework='cloud-build', AutoMLOps will use:

  3. if use_ci=True and orchestration_framework='kfp' and deployment_framework='cloud-build', AutoMLOps will use:

  4. if use_ci=True and pipeline_job_submission_service_type='cloud-functions', AutoMLOps will use:

  5. if use_ci=True and pipeline_job_submission_service_type='cloud-run', AutoMLOps will use:

  6. if use_ci=True and schedule_pattern is specified, AutoMLOps will use:

  7. if use_ci=True and source_repo_type='cloud-source-repositories', AutoMLOps will use:

  8. if use_ci=True and setup_model_monitoring=True, AutoMLOps will use:

APIs & IAM

Based on the above user selection, AutoMLOps will enable up to the following APIs during the provision step:

AutoMLOps will create the following service account and update IAM permissions during the provision step:

  1. Pipeline Runner Service Account (defaults to: vertex-pipelines@PROJECT_ID.iam.gserviceaccount.com). Roles added:
    • roles/aiplatform.user
    • roles/artifactregistry.reader
    • roles/bigquery.user
    • roles/bigquery.dataEditor
    • roles/iam.serviceAccountUser
    • roles/storage.admin
    • roles/cloudfunctions.admin

Prechecks and Warnings

AutoMLOps provides a number of optional prechecks and warnings to provide visibility to the user into what IAM permissions are required to run certain operations.

  1. AutoMLOps.provision(...hide_warnings=False): This will check installation versions and account permissions to determine if the current account has the permissions to provision successfully.
  2. AutoMLOps.deploy(...hide_warnings=False): This will check installation versions and account permissions to determine if the current account has the permissions to deploy successfully.
  3. AutoMLOps.deploy(...precheck=True): This will check for the necessary infrastructure to deploy successfully (e.g. does the specified artifact registry exist, does the specified storage bucket exist, etc.)
  4. AutoMLOps.monitor(...hide_warnings=False): This will check installation versions and account permissions to determine if the current account has the permissions to create model monitoring jobs successfully.

Code Generation Options

AutoMLOps CI/CD options:

  1. use_ci: Bool that specifies whether to execute using generated files and scripts locally or use cloud CI/CD workflow. Defaults to False. See CI/CD Workflow

Required parameters:

  1. project_id: str
  2. pipeline_params: dict

Optional parameters (defaults shown):

  1. artifact_repo_location: str = 'us-central1'
  2. artifact_repo_name: str = f'{naming_prefix}-artifact-registry'
  3. artifact_repo_type: str = 'artifact-registry'
  4. base_image: str = 'python:3.9-slim'
  5. build_trigger_location: str = 'us-central1'
  6. build_trigger_name: str = f'{naming_prefix}-build-trigger'
  7. custom_training_job_specs: list[dict] = None
  8. deployment_framework: str = 'cloud-build'
  9. naming_prefix: str = 'automlops-default-prefix'
  10. orchestration_framework: str = 'kfp'
  11. pipeline_job_runner_service_account: str = f'vertex-pipelines@{project_id}.iam.gserviceaccount.com'
  12. pipeline_job_submission_service_location: str = 'us-central1'
  13. pipeline_job_submission_service_name: str = f'{naming_prefix}-job-submission-svc'
  14. pipeline_job_submission_service_type: str = 'cloud-functions'
  15. project_number: str = None
  16. provision_credentials_key: str = None
  17. provisioning_framework: str = 'gcloud'
  18. pubsub_topic_name: str = f'{naming_prefix}-queueing-svc'
  19. schedule_location: str = 'us-central1'
  20. schedule_name: str = f'{naming_prefix}-schedule'
  21. schedule_pattern: str = 'No Schedule Specified'
  22. setup_model_monitoring: Optional[bool] = False
  23. source_repo_branch: str = 'automlops'
  24. source_repo_name: str = f'{naming_prefix}-repository'
  25. source_repo_type: str = 'cloud-source-repositories'
  26. storage_bucket_location: str = 'us-central1'
  27. storage_bucket_name: str = f'{project_id}-{naming_prefix}-bucket'
  28. use_ci: bool = False
  29. vpc_connector: str = 'No VPC Specified'
  30. workload_identity_pool: str = None
  31. workload_identity_provider: str = None
  32. workload_identity_service_account: str = None

Parameter Options:

A description of the parameters is below:

AutoMLOps will generate the resources specified by these parameters (e.g. Artifact Registry, Cloud Source Repo, etc.). If use_ci is set to True, AutoMLOps will turn the outputted AutoMLOps/ directory into a Git repo and use it for the source repo. If a cron formatted str is given as an arg for schedule_pattern then it will set up a Cloud Schedule to run accordingly. If setup_model_monitoring is set to true, a model_monitoring/ directory will be created and a monitoring section will be added to config/defaults.yaml with empty values. These values are then set by running AutoMLOps.monitor().

Generating Code

AutoMLOps generates code that is compatible with kfp>=2.0.0. Upon running AutoMLOps.generate(project_id='project-id', pipeline_params=pipeline_params, setup_model_monitoring=True, use_ci=True), a series of directories will be generated automatically:

.
├── components                                     : Custom vertex pipeline components.
    ├──component_base                              : Contains all the python files, Dockerfile and requirements.txt
        ├── Dockerfile                             : Dockerfile containing all the python files for the components.
        ├── requirements.txt                       : Package requirements for all the python files for the components.
        ├── src                                    : Python source code directory.
            ├──component_a.py                      : Python file containing code for the component.
            ├──...(for each component)
    ├──component_a                                 : Components specs generated using AutoMLOps
        ├── component.yaml                         : Component yaml spec, acts as an I/O wrapper around the Docker container.
    ├──...(for each component)
├── configs                                        : Configurations for defining vertex ai pipeline and MLOps infra.
    ├── defaults.yaml                              : Runtime configuration variables.
├── images                                         : Custom container images for training models (optional).
├── pipelines                                      : Vertex ai pipeline definitions.
    ├── pipeline.py                                : Full pipeline definition; compiles pipeline spec and uploads to GCS.
    ├── pipeline_runner.py                         : Sends a PipelineJob to Vertex AI.
    ├── requirements.txt                           : Package requirements for running pipeline.py.
    ├── runtime_parameters                         : Variables to be used in a PipelineJob.
        ├── pipeline_parameter_values.json         : Json containing pipeline parameters.
├── provision                                      : Provision configurations and details.
    ├── provision_resources.sh                     : Provisions the necessary infra to run the MLOps pipeline.
    ├── provisioning_configs                       : (Optional) Relevant terraform/Pulumi config files for provisioning infa.
├── scripts                                        : Scripts for manually triggering the cloud run service.
    ├── build_components.sh                        : Submits a Cloud Build job that builds and pushes the components to the registry.
    ├── build_pipeline_spec.sh                     : Compiles the pipeline specs.
    ├── run_pipeline.sh                            : Submit the PipelineJob to Vertex AI.
    ├── run_all.sh                                 : Builds components, compiles pipeline specs, and submits the PipelineJob.
    ├── publish_to_topic.sh                        : Publishes a message to a Pub/Sub topic to invoke the pipeline job submission service.
    ├── create_model_monitoring_job.sh             : Creates or updated a Vertex AI model monitoring job for a given deployed model endpoint.
├── model_monitoring                               : Code for building and maintaining model monitoring jobs.
    ├── requirements.txt                           : Package requirements for creating and updating model monitoring jobs.
    ├── monitor.py                                 : Creates a ModelDeploymentMonitoringJob and optionally creates a Log Sink for automatic retraining.
├── services                                       : MLOps services related to continuous training.
    ├── submission_service                         : REST API service used to submit pipeline jobs to Vertex AI.
        ├── Dockerfile                             : Dockerfile for running the REST API service.
        ├── requirements.txt                       : Package requirements for the REST API service.
        ├── main.py                                : Python REST API source code.
├── README.md                                      : Readme markdown file describing the contents of the generated directories.
└── Build config yaml                              : Build configuration file for building custom components.

Provisioning

AutoMLOps currently provides 2 primary options for provisioning infrastructure: gcloud and terraform. In the diagram below dashed boxes show areas users can select and customize their tooling.

CICD

Deployment

Cloud Continuous Integration and Continuous Deployment Workflow

If use_ci=True, AutoMLOps will generate and use a fully featured CI/CD environment for the pipeline. Otherwise, it will use the local scripts to build and run the pipeline. In the diagrams below dashed boxes show areas users can select and customize their tooling.

Cloud Build option:

CICD

Github Actions option:

CICD

Gitlab CI option:

CICD

Bitbucket Pipelines option:

CICD

Model Monitoring

To use AutoMLOps for setting up and creating model monitoring jobs, the setup_model_monitoring parameter must be set to True during the generate step. This will create the necessary files under AutoMLOps/scripts and AutoMLOps/model_monitoring. From there, configurations can be set using AutoMLOps.monitor. This operation requires a functioning Vertex AI model endpoint, and the name of the endpoint's prediction column. Monitoring can be set to check for anomalies in 2 ways:

  1. Data Drift: This compares incoming feature distributions to past feature distributions that the model endpoint has seen, within a given window of time.
  2. Data Skew: This compares incoming feature distributions to feature distributions within the training dataset to check for training/serving skew.

At least one of these monitoring types must be used. To specify drift and skew, provide a dictionary of feature names and their threshold values to the drift_thresholds and skew_thresholds parameters.

When an anomaly is detected, actions can be taken in 2 ways:

  1. Generate email alerts: This will send an email to the specified email(s) informing the user of a detected data drift or skew.
  2. Automatically retrain the model: This will send anomaly alerts to the Pub/Sub Queueing service and trigger a retraining of the model, with the parameters specified, when a data drift or skew is detected.

These can be configured using the alert_emails and auto_retraining_params parameters, which are left null by default. If auto_retraining_params are specified, AutoMLOps.monitor will set up a Log Sink to connect the Vertex AI Model Monitoring Job to the Pub/Sub Queueing service, and filter on only ModelMonitoringJob anomalies.

Before running AutoMLOps.monitor(), be sure to install the package dependencies using the following command: pip3 install -r AutoMLOps/model_monitoring/requirements.txt --user

AutoMLOps.monitor() parameter list:

Required parameters:

  1. target_field: str
  2. model_endpoint: str

Optional parameters (defaults shown):

  1. alert_emails: list = None
  2. auto_retraining_params: dict = None
  3. drift_thresholds: dict = None
  4. hide_warnings: bool = True
  5. job_display_name: str = f'{naming_prefix}-model-monitoring-job'
  6. monitoring_interval: int = 1
  7. monitoring_location: str = 'us-central1'
  8. sample_rate: float = 0.8
  9. skew_thresholds: dict = None
  10. training_dataset: str = None

A description of the parameters is below:

Model Monitoring Diagram

CICD

Other Customizations

Using Github Actions:

To use Github Actions integration, you must first have a Workload Identity Federation set up properly. You must use a pre-existing Github repo, and you must also have already set up and registered your ssh keys with your Github Repo. If you meet all of these prerequisites, you can use github actions as follows:

AutoMLOps.generate(project_id=PROJECT_ID,
                   pipeline_params=pipeline_params,
                   use_ci=True,
                   deployment_framework='github-actions',
                   project_number='<project_number>',
                   source_repo_type='github',
                   source_repo_name='source/repo/string',
                   workload_identity_pool='identity_pool_string',
                   workload_identity_provider='identity_provider_string',
                   workload_identity_service_account='workload_identity_sa')

Set scheduled run:

Use the schedule_pattern parameter to specify a cron job schedule to run the pipeline job on a recurring basis.

schedule_pattern = '0 */12 * * *'

Use Vertex AI Experiments:

To use Vertex AI Experiments, include key-value pair for vertex_experiment_tracking_name in your pipeline parameters dictionary. An experiment will be created if one does not already exist with the specified name.

pipeline_params = {
    'project_id': PROJECT_ID,
    'region': 'us-central1',
    'vertex_experiment_tracking_name': 'my-experiment-name'
}
AutoMLOps.generate(project_id=PROJECT_ID,
                   pipeline_params=pipeline_params)

Set pipeline compute resources:

Use the base_image and custom_training_job_specs parameter to specify resources for any custom component in the pipeline.

base_image = 'us-docker.pkg.dev/vertex-ai/training/tf-gpu.2-11.py310:latest',
custom_training_job_specs = [{
    'component_spec': 'train_model',
    'display_name': 'train-model-accelerated',
    'machine_type': 'a2-highgpu-1g',
    'accelerator_type': 'NVIDIA_TESLA_A100',
    'accelerator_count': 1
}]

Use a VPC connector:

Use the vpc_connector parameter to specify a vpc connector.

vpc_connector = 'example-vpc-connector'

Specify package versions:

Use the packages_to_install parameter of @AutoMLOps.component to explicitly specify packages and versions.

@AutoMLOps.component(
    packages_to_install=[
        "google-cloud-bigquery==2.34.4",
        "pandas",
        "pyarrow",
        "db_dtypes"
    ]
)
def create_dataset(
    bq_table: str,
    data_path: str,
    project_id: str
):
...

Contributors

Sean Rastatter: Tech Lead

Tony DiLoreto: Project Manager

Allegra Noto: Senior Project Engineer

Ahmad Khan: Engineer

Jesus Orozco: Engineer

Erin Horning: Engineer

Alex Ho: Engineer

Kyle Sorensen: Engineer

Matt Sokoloff: Engineer

Andrew Chasin: Engineer

Atulan Zaman: Use Case Development

Nelly Wilson: Use Case Development

Disclaimer

This is not an officially supported Google product.

Copyright 2024 Google LLC. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.