search

GoogleCloudPlatform / mlops-with-vertex-ai

An end-to-end example of MLOps on Google Cloud using TensorFlow, TFX, and Vertex AI
Apache License 2.0
341 stars 117 forks source link

Missing gcloud command or terraform code to create cloud build triggers #19

Open hilliao opened 2 years ago

hilliao commented 2 years ago

Users like to have a way to create cloud build triggers in the Terraform folder or with gcloud commands. I found tensorflow transform is using Apache beam runner in the following cloud build file. 

# Compile the pipeline.
- name: '$_CICD_IMAGE_URI'
  entrypoint: 'python'
  args: ['build/utils.py',
          '--mode', 'compile-pipeline',
          '--pipeline-name', '$_PIPELINE_NAME'
          ]
  dir: 'mlops-with-vertex-ai'
  env: 
  - 'PROJECT=$_PROJECT'  
  - 'REGION=$_REGION'
  - 'MODEL_DISPLAY_NAME=$_MODEL_DISPLAY_NAME'
  - 'DATASET_DISPLAY_NAME=$_DATASET_DISPLAY_NAME'  
  - 'GCS_LOCATION=$_GCS_LOCATION' 
  - 'TFX_IMAGE_URI=$_TFX_IMAGE_URI' 
  - 'BEAM_RUNNER=$_BEAM_RUNNER'
  - 'TRAINING_RUNNER=$_TRAINING_RUNNER'
  id: 'Compile Pipeline'
  waitFor: ['Local Test E2E Pipeline']

So far I gather an example from a notebook's section. But I can't find how to create the cloud build trigger.

  1. Run the training pipeline using Vertex Pipelines Set the pipeline configurations for the Vertex AI run os.environ["DATASET_DISPLAY_NAME"] = DATASET_DISPLAY_NAME os.environ["MODEL_DISPLAY_NAME"] = MODEL_DISPLAY_NAME os.environ["PIPELINE_NAME"] = PIPELINE_NAME os.environ["PROJECT"] = PROJECT os.environ["REGION"] = REGION os.environ["GCS_LOCATION"] = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}" os.environ["TRAIN_LIMIT"] = "85000" os.environ["TEST_LIMIT"] = "15000" os.environ["BEAM_RUNNER"] = "DataflowRunner" os.environ["TRAINING_RUNNER"] = "vertex" os.environ["TFX_IMAGE_URI"] = f"gcr.io/{PROJECT}/{DATASET_DISPLAY_NAME}:{VERSION}" os.environ["ENABLE_CACHE"] = "1"