[Question] Is there a way to provision multiple environments (dev, prd)

[jogando]

Hi, Usually when we work in GCP projects for customers we have multiple environments: development, production and so on. Is there a recommended way of provisioning multiple environments using AutoMLOps? Ideally we would have a "dev" branch with a separte environment (GCP resources + pipeline), we would make our changes in this branch and once everything is tested we would create PR for merging "dev" with the "main" branch in order for deploying the changes to Production.

Thanks!

[jogando]

AutoMLOps does not supports multiple environments when Cloud Source Repositories is used with Terraform, since Terraform will try to create the same repository (and fail) for each environment. The following approach was implemented using Terraform and Github Actions

ENVIRONMENT="dev"

pipeline_params = {
    "project": PROJECT_ID,
    "location": REGION,
    "environment": ENVIRONMENT
}

AutoMLOps.generate(
    project_id=PROJECT_ID,
    pipeline_params=pipeline_params,
    use_ci=True,
    naming_prefix=f"{PIPELINE_NAME}-{ENVIRONMENT}",
    schedule_pattern='59 11 * * 0', # retrain every Sunday at Midnight,
    pipeline_job_runner_service_account=f"{PIPELINE_NAME}-{ENVIRONMENT}-runner@{PROJECT_ID}.iam.gserviceaccount.com",
    provisioning_framework="terraform",
    source_repo_name=SOURCE_REPO_NAME,
    source_repo_type="github",
    deployment_framework="github-actions",
    project_number=PROJECT_NUMBER,
    workload_identity_provider=WORKLOAD_IDENTITY_PROVIDER,
    workload_identity_service_account=WORKLOAD_IDENTITY_SERVICE_ACCOUNT,
    workload_identity_pool=WORKLOAD_IDENTITY_POOL,
    source_repo_branch=ENVIRONMENT
)

AutoMLOps.provision(hide_warnings=False)

AutoMLOps.deploy(precheck=False, hide_warnings=False)

Once the DEV environment is provisioned, we need to perform the following to provision the PRD environment:

# Remove the 'provision/state_bucket' terraform files from the other environment
! rm ./AutoMLOps/provision/state_bucket/terraform.tfstate
! rm ./AutoMLOps/provision/state_bucket/.terraform.lock.hcl
! rm -rf ./AutoMLOps/provision/state_bucket/.terraform

# Remove the 'provision/environment' terraform files from the other environment

! rm ./AutoMLOps/provision/environment/.terraform.lock.hcl
! rm -rf ./AutoMLOps/provision/environment/.terraform

ENVIRONMENT="prd"
! git -C ./AutoMLOps checkout -b $ENVIRONMENT

pipeline_params = {
    "project": PROJECT_ID,
    "location": REGION,
    "environment": ENVIRONMENT
}

AutoMLOps.generate(
    project_id=PROJECT_ID,
    pipeline_params=pipeline_params,
    use_ci=True,
    naming_prefix=f"{PIPELINE_NAME}-{ENVIRONMENT}",
    schedule_pattern='59 11 * * 0', # retrain every Sunday at Midnight,
    pipeline_job_runner_service_account=f"{PIPELINE_NAME}-{ENVIRONMENT}-runner@{PROJECT_ID}.iam.gserviceaccount.com",
    provisioning_framework="terraform",
    source_repo_name=SOURCE_REPO_NAME,
    source_repo_type="github",
    deployment_framework="github-actions",
    project_number=PROJECT_NUMBER,
    workload_identity_provider=WORKLOAD_IDENTITY_PROVIDER,
    workload_identity_service_account=WORKLOAD_IDENTITY_SERVICE_ACCOUNT,
    workload_identity_pool=WORKLOAD_IDENTITY_POOL,
    source_repo_branch=ENVIRONMENT
)

AutoMLOps.provision(hide_warnings=False)

AutoMLOps.deploy(precheck=False, hide_warnings=False)