microsoft / MLOpsPython

MLOps using Azure ML Services and Azure DevOps
MIT License
1.2k stars 1.09k forks source link

Pipeline job failure during "Azure resource group deployment" #255

Closed SwathiMystery closed 4 years ago

SwathiMystery commented 4 years ago

When I run the pipeline from iac-create-environment-pipeline.yml, the job fails at Azure resource group deployment task

I have followed the documentation and executed the steps till run the pipeline. Even if I run for the first time with the new BASE_NAME in the variable group, I get the following error.

The failure log is as follows:

2020-04-07T13:03:08.8411809Z ##[section]Starting: Deploy MLOps resources to Azure 2020-04-07T13:03:08.8422847Z ============================================================================== 2020-04-07T13:03:08.8423437Z Task : Azure resource group deployment 2020-04-07T13:03:08.8423846Z Description : Deploy an Azure Resource Manager (ARM) template to a resource group and manage virtual machines 2020-04-07T13:03:08.8424240Z Version : 2.165.0 2020-04-07T13:03:08.8424506Z Author : Microsoft Corporation 2020-04-07T13:03:08.8424920Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment 2020-04-07T13:03:08.8425384Z ============================================================================== 2020-04-07T13:03:09.4226512Z Checking if the following resource group exists: mlopstest-RG. 2020-04-07T13:03:09.7095675Z Resource group exists: false. 2020-04-07T13:03:09.7098379Z Creating resource Group: mlopstest-RG 2020-04-07T13:03:10.2049353Z Resource Group created successfully. 2020-04-07T13:03:10.2054292Z Creating deployment parameters. 2020-04-07T13:03:10.2278426Z The detected encoding for file '/home/vsts/work/1/s/environment_setup/arm-templates/cloud-environment.json' is 'utf-8' 2020-04-07T13:03:10.4724660Z Starting template validation. 2020-04-07T13:03:10.4771834Z Deployment name is cloud-environment-20200407-130310-da30 2020-04-07T13:03:11.7864695Z There were errors in your deployment. Error code: InvalidTemplateDeployment. 2020-04-07T13:03:11.7902270Z ##[error]The template deployment 'cloud-environment-20200407-130310-da30' is not valid according to the validation procedure. The tracking id is '51a3cd8c-1a6c-4116-958e-929959923704'. See inner errors for details. 2020-04-07T13:03:11.7912733Z ##[error]Details: 2020-04-07T13:03:11.7914716Z ##[error]PreflightValidationCheckFailed: Preflight validation failed. Please refer to the details for the specific errors. [{"code":"StorageAccountAlreadyTaken","target":"mlopstestamlsa","message":"The storage account named mlopstestamlsa is already taken."}] 2020-04-07T13:03:11.7917418Z ##[warning]Validation errors were found in the Azure Resource Manager template. This can potentially cause template deployment to fail. Task failed while creating or updating the template deployment.. Please follow https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-syntax 2020-04-07T13:03:11.7918616Z Starting Deployment. 2020-04-07T13:03:11.7919158Z Deployment name is cloud-environment-20200407-130310-da30 2020-04-07T13:03:27.5102034Z There were errors in your deployment. Error code: DeploymentFailed. 2020-04-07T13:03:27.5113153Z ##[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details. 2020-04-07T13:03:27.5117792Z ##[error]Details: 2020-04-07T13:03:27.5122528Z ##[error]Conflict: { "error": { "code": "StorageAccountAlreadyTaken", "message": "The storage account named mlopstestamlsa is already taken." } } 2020-04-07T13:03:27.5126804Z ##[error]Conflict: { "error": { "code": "VaultAlreadyExists", "message": "The name 'mlopstest-AML-KV' is already in use." } } 2020-04-07T13:03:27.5131991Z ##[error]Conflict: { "error": { "code": "AlreadyInUse", "message": "The registry DNS name mlopstestamlcr.azurecr.io is already in use. You can check if the name is already claimed using following API: https://docs.microsoft.com/en-us/rest/api/containerregistry/registries/checknameavailability" }, "status": "Failed" } 2020-04-07T13:03:27.5135708Z ##[error]Task failed while creating or updating the template deployment. 2020-04-07T13:03:27.5185871Z ##[section]Finishing: Deploy MLOps resources to Azure

Any help is very much appreciated.

Thanks!

eedorenko commented 4 years ago

The BASE_NAME should be unique across Azure cloud. "mlopstest" doesn't seem like that. According to the log you have attached, there are already resources with the names "mlopstestamlsa", "mlopstest-AML-KV", mlopstestamlcr, etc.

SwathiMystery commented 4 years ago

Thank you for your reply.

I did try that with different BASE_NAME (unique). Even if I created everything afresh with a new name, the first run fails as above.

Thank you!

dtzar commented 4 years ago

Please post the logs after modifying it to a new unique name.

j-so commented 4 years ago

@SwathiMystery Are you still having this issue? Feel free to reopen if you still need assistance here.