Open codemohsmohammad789 opened 1 year ago
Ideally this tasks fails while creating conda environment., and there is no proper error while creating as well. please help to resolve the issue
======Starting Image Build on Serverless Compute====== The run ID for the image build on serverless compute is imgbldrun_725f1f0 Additional logs for the run: https://ml.azure.com/experiments/id/prepare_image/runs/imgbldrun_725f1f0?wsid=/subscriptions/eae29237-49ec-42cc-82e7-dc1125499c88/resourcegroups/rg-sandpox-amrc/workspaces/sandbox-amrc-aml&tid=6f2b364d-3ab6-42e1-a351-e4d69eb92fe9 2024-09-05T15:26:38: Logging into Docker registry: d30f8cdd495e4db28bb266fa51695ba3.azurecr.io 2024-09-05T15:26:38: WARNING! Using --password via the CLI is insecure. Use --password-stdin. 2024-09-05T15:26:38: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. 2024-09-05T15:26:38: Configure a credential helper to remove this warning. See 2024-09-05T15:26:38: https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-09-05T15:26:38: Login Succeeded
2024-09-05T15:26:38: Running: ['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'd30f8cdd495e4db28bb266fa51695ba3.azurecr.io/azureml/azureml_134293382750c1bb119025659155ea48', '-t', 'd30f8cdd495e4db28bb266fa51695ba3.azurecr.io/azureml/azureml_134293382750c1bb119025659155ea48:1'] 2024-09-05T15:26:38: DEPRECATED: The legacy builder is deprecated and will be removed in a future release. 2024-09-05T15:26:38: Install the buildx component to build images with BuildKit: 2024-09-05T15:26:38: https://docs.docker.com/go/buildx/
2024-09-05T15:26:38: Sending build context to Docker daemon 71.17kB
2024-09-05T15:26:38: Step 1/17 : FROM mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04:20210301.v1@sha256:000d6c43f606ceaa67983790ca95c70fd741c364d8c2e3217a11d775b99741df
2024-09-05T15:26:38: mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04:20210301.v1@sha256:000d6c43f606ceaa67983790ca95c70fd741c364d8c2e3217a11d775b99741df: Pulling from azureml/intelmpi2018.3-ubuntu16.04
2024-09-05T15:26:38: 4007a89234b4: Pulling fs layer
2024-09-05T15:26:38: 5dfa26c6b9c9: Pulling fs layer
2024-09-05T15:26:38: 0ba7bf18aa40: Pulling fs layer
2024-09-05T15:26:38: 4c6ec688ebe3: Pulling fs layer
2024-09-05T15:26:38: 574f361512d6: Pulling fs layer
2024-09-05T15:26:38: db4d1e2d7079: Pulling fs layer
2024-09-05T15:26:38: e544ee0f522d: Pulling fs layer
2024-09-05T15:26:38: c655136086be: Pulling fs layer
2024-09-05T15:26:38: 2ec37f44090c: Pulling fs layer
2024-09-05T15:26:38: 5fba3bd4a2c4: Pulling fs layer
2024-09-05T15:26:38: 7e0ea9d0a1ab: Pulling fs layer
2024-09-05T15:26:38: da005f826951: Pulling fs layer
2024-09-05T15:26:38: 4c6ec688ebe3: Waiting
2024-09-05T15:26:38: 574f361512d6: Waiting
2024-09-05T15:26:38: db4d1e2d7079: Waiting
2024-09-05T15:26:38: e544ee0f522d: Waiting
2024-09-05T15:26:38: 7e0ea9d0a1ab: Waiting
2024-09-05T15:26:38: c655136086be: Waiting
2024-09-05T15:26:38: da005f826951: Waiting
2024-09-05T15:26:38: 2ec37f44090c: Waiting
2024-09-05T15:26:38: 5fba3bd4a2c4: Waiting
2024-09-05T15:26:38: 0ba7bf18aa40: Verifying Checksum
2024-09-05T15:26:38: 0ba7bf18aa40: Download complete
2024-09-05T15:26:38: 5dfa26c6b9c9: Download complete
2024-09-05T15:26:38: 4c6ec688ebe3: Verifying Checksum
2024-09-05T15:26:38: 4c6ec688ebe3: Download complete
2024-09-05T15:26:39: db4d1e2d7079: Verifying Checksum
2024-09-05T15:26:39: db4d1e2d7079: Download complete
2024-09-05T15:26:39: 574f361512d6: Verifying Checksum
2024-09-05T15:26:39: 574f361512d6: Download complete
2024-09-05T15:26:40: e544ee0f522d: Verifying Checksum
2024-09-05T15:26:40: e544ee0f522d: Download complete
2024-09-05T15:26:40: 4007a89234b4: Verifying Checksum
2024-09-05T15:26:40: 4007a89234b4: Download complete
2024-09-05T15:26:40: 5fba3bd4a2c4: Verifying Checksum
2024-09-05T15:26:40: 5fba3bd4a2c4: Download complete
2024-09-05T15:26:41: 7e0ea9d0a1ab: Verifying Checksum
2024-09-05T15:26:41: 7e0ea9d0a1ab: Download complete
2024-09-05T15:26:41: da005f826951: Verifying Checksum
2024-09-05T15:26:41: da005f826951: Download complete
2024-09-05T15:26:43: 2ec37f44090c: Verifying Checksum
2024-09-05T15:26:43: 2ec37f44090c: Download complete
2024-09-05T15:26:43: 4007a89234b4: Pull complete
2024-09-05T15:26:44: c655136086be: Verifying Checksum
2024-09-05T15:26:44: c655136086be: Download complete
2024-09-05T15:26:47: 5dfa26c6b9c9: Pull complete
2024-09-05T15:26:47: 0ba7bf18aa40: Pull complete
2024-09-05T15:26:47: 4c6ec688ebe3: Pull complete
2024-09-05T15:26:55: 574f361512d6: Pull complete
2024-09-05T15:26:56: db4d1e2d7079: Pull complete
2024-09-05T15:27:00: e544ee0f522d: Pull complete
2024-09-05T15:27:05: c655136086be: Pull complete
2024-09-05T15:27:11: 2ec37f44090c: Pull complete
2024-09-05T15:27:11: 5fba3bd4a2c4: Pull complete
2024-09-05T15:27:12: 7e0ea9d0a1ab: Pull complete
2024-09-05T15:27:12: da005f826951: Pull complete
2024-09-05T15:27:12: Digest: sha256:000d6c43f606ceaa67983790ca95c70fd741c364d8c2e3217a11d775b99741df
2024-09-05T15:27:12: Status: Downloaded newer image for mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04:20210301.v1@sha256:000d6c43f606ceaa67983790ca95c70fd741c364d8c2e3217a11d775b99741df
2024-09-05T15:27:12: ---> c942df5ba5d0
2024-09-05T15:27:12: Step 2/17 : USER root
2024-09-05T15:27:12: ---> Running in aa672599f5bc
2024-09-05T15:27:13: Removing intermediate container aa672599f5bc
2024-09-05T15:27:13: ---> 16f637ab12ee
2024-09-05T15:27:13: Step 3/17 : RUN mkdir -p $HOME/.cache
2024-09-05T15:27:13: ---> Running in 58efac0e7aec
2024-09-05T15:27:15: Removing intermediate container 58efac0e7aec
2024-09-05T15:27:15: ---> 0b9557f3ff21
2024-09-05T15:27:15: Step 4/17 : WORKDIR /
2024-09-05T15:27:15: ---> Running in 3ee153c2e13f
2024-09-05T15:27:16: Removing intermediate container 3ee153c2e13f
2024-09-05T15:27:16: ---> 856824b7accb
2024-09-05T15:27:16: Step 5/17 : COPY azureml-environment-setup/99brokenproxy /etc/apt/apt.conf.d/
2024-09-05T15:27:17: ---> be010d5a63a9
2024-09-05T15:27:17: Step 6/17 : RUN if dpkg --compare-versions conda --version | grep -oE '[^ ]+$'
lt 4.4.11; then conda install conda==4.4.11; fi
2024-09-05T15:27:17: ---> Running in 8de29dbf8643
2024-09-05T15:27:20: Removing intermediate container 8de29dbf8643
2024-09-05T15:27:20: ---> 736f324cdb84
2024-09-05T15:27:20: Step 7/17 : COPY azureml-environment-setup/mutated_conda_dependencies.yml azureml-environment-setup/mutated_conda_dependencies.yml
2024-09-05T15:27:21: ---> 26d618b1d31e
2024-09-05T15:27:21: Step 8/17 : RUN ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_f7d0f75c0d0e79ca21c80a06521c5814 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name pycache -exec rm -rf {} + && ldconfig
2024-09-05T15:27:21: ---> Running in 828ca357583d
2024-09-05T15:27:33: Collecting package metadata (repodata.json): ...working... done
2024-09-05T15:27:33: Solving environment: ...working... done
2024-09-05T15:27:33: Downloading and Extracting Packages 2024-09-05T15:27:34: ncurses-6.0 | 781 KB | ########## | 100% 2024-09-05T15:27:34: zlib-1.2.13 | 111 KB | ########## | 100% 2024-09-05T15:27:34: wheel-0.37.1 | 33 KB | ########## | 100% 2024-09-05T15:27:34: sqlite-3.23.1 | 808 KB | ########## | 100% 2024-09-05T15:27:35: python-3.6.2 | 23.6 MB | ########## | 100% 2024-09-05T15:27:35: xz-5.4.6 | 643 KB | ########## | 100% 2024-09-05T15:27:36: tk-8.6.14 | 3.4 MB | ########## | 100% 2024-09-05T15:27:36: readline-7.0 | 848 KB | ########## | 100% 2024-09-05T15:27:36: certifi-2021.5.30 | 139 KB | ########## | 100% 2024-09-05T15:27:36: libgcc-ng-11.2.0 | 5.3 MB | ########## | 100% 2024-09-05T15:27:37: ca-certificates-2024 | 127 KB | ########## | 100% 2024-09-05T15:27:37: libedit-3.1 | 151 KB | ########## | 100% 2024-09-05T15:27:37: libgomp-11.2.0 | 474 KB | ########## | 100% 2024-09-05T15:27:37: libstdcxx-ng-11.2.0 | 4.7 MB | ########## | 100% 2024-09-05T15:27:37: _libgcc_mutex-0.1 | 3 KB | ########## | 100% 2024-09-05T15:27:38: setuptools-58.0.4 | 788 KB | ########## | 100% 2024-09-05T15:27:38: libffi-3.2.1 | 48 KB | ########## | 100% 2024-09-05T15:27:38: openssl-1.0.2u | 2.2 MB | ########## | 100% 2024-09-05T15:27:38: pip-21.2.2 | 1.8 MB | ########## | 100% 2024-09-05T15:27:38: _openmp_mutex-5.1 | 21 KB | ########## | 100% 2024-09-05T15:27:39: Preparing transaction: ...working... done 2024-09-05T15:27:40: Verifying transaction: ...working... done 2024-09-05T15:27:41: Executing transaction: ...working... done
I am trying to run the train pipeline, I get an error everytime I am unable to figure it out. In the POn ST method link when I run it get an error unsupported AI verson 1.0 error. Please help I have been trying to run the invoke Ml Pipeline from a long time
Job preparation parameters Variables: ACI_DEPLOYMENT_NAME: $[ variablegroups.devopsforai-aml-vg.ACI_DEPLOYMENT_NAME ] AZURE_RM_SVC_CONNECTION: $[ variablegroups.devopsforai-aml-vg.AZURE_RM_SVC_CONNECTION ] BASE_NAME: $[ variablegroups.devopsforai-aml-vg.BASE_NAME ] LOCATION: $[ variablegroups.devopsforai-aml-vg.LOCATION ] RESOURCE_GROUP: $[ variablegroups.devopsforai-aml-vg.RESOURCE_GROUP ] WORKSPACE_NAME: $[ variablegroups.devopsforai-aml-vg.WORKSPACE_NAME ] WORKSPACE_SVC_CONNECTION: $[ variablegroups.devopsforai-aml-vg.WORKSPACE_SVC_CONNECTION ] AMLPIPELINE_ID: Parsing expression: <dependencies.Get_Pipeline_ID.outputs['getpipelineid.AMLPIPELINEID']> Evaluating: dependencies['Get_Pipeline_ID']['outputs']['getpipelineid.AMLPIPELINEID'] Result: '3ae98478-5ffa-414a-8363-a3e3b3af42db' ContinueOnError: False TimeoutInMinutes: 0 CancelTimeoutInMinutes: 5 Expand: MaxConcurrency: 0 MaxConcurrency: 0 ============================================================================== Task : Invoke ML pipeline Description : Invoke Azure Machine Learning published pipelines. Version : 0.1.90 Author : ms-air-aiagility Help URL : https://marketplace.visualstudio.com/items?itemName=ms-air-aiagility.vss-services-azureml&ssr=false#overview ============================================================================== 2023-07-12T11:06:47.6918609Z POST https://centralus.aether.ms/api/v1.0/subscriptions/19c1eed4-f0da-4a7c-8bd9-e82e6700f2b7/resourceGroups/mmlops-RG/providers/Microsoft.MachineLearningServices/workspaces/mosml008/PipelineRuns/PipelineSubmit/3ae98478-5ffa-414a-8363-a3e3b3af42db Request body: { "AzDevOpsCallBackInfo": { "JobId": "77aee9c5-85bc-59ae-fa90-c3f20f9c111e", "HubName": "build", "TaskInstanceId": "7ffef4a3-944d-544a-2764-6fcc510d84c9", "PlanId": "6bebf0d7-465c-4c93-ad8e-49e907470d26", "ProjectId": "0c53d160-630b-4a98-aa55-51747305200b", "PlanUri": "https://dev.azure.com/mohsmohammad7890561/", "Auth": }, "ExperimentName": "mlopspython", "RunSource": "PipelineRun", "ParameterAssignments": { "model_name": "diabetes_regression_model.pkl" }, "tags": { "BuildId": "36", "BuildUri": "https://dev.azure.com/mohsmohammad7890561/MLOPS/_build/results?buildId=36" }, "StepTags": { "BuildId": "36", "BuildUri": "https://dev.azure.com/mohsmohammad7890561/MLOPS/_build/results?buildId=36" } } Response Code: OK Response: { "Description": null, "Status": { "StatusCode": 0, "StatusDetail": null, "CreationTime": "2023-07-12T11:06:47.1409291Z", "EndTime": null }, "GraphId": "98f140f2-05f8-4064-9915-5de767a25c61", "IsSubmitted": false, "HasErrors": false, "HasWarnings": false, "UploadState": 0, "ParameterAssignments": { "model_name": "diabetes_regression_model.pkl", "dataset_version": "latest", "data_file_path": "none", "caller_run_id": "none" }, "DataPathAssignments": {}, "DataSetDefinitionValueAssignments": {}, "AssetOutputSettingsAssignments": {}, "RunHistoryExperimentName": "mlopspython", "DisplayName": null, "PipelineRunId": "8ed03654-914e-4c2a-b633-3090bdf94669", "PipelineId": "3ae98478-5ffa-414a-8363-a3e3b3af42db", "PipelineEndpointId": null, "RunSource": "PipelineRun", "RunType": 0, "TotalRunSteps": 3, "ScheduleId": null, "RunUrl": "https://ml.azure.com/experiments/mlopspython/runs/8ed03654-914e-4c2a-b633-3090bdf94669?tid=39ac8b47-69b1-4db6-b731-b6591aee89d3&wsid=/subscriptions/19c1eed4-f0da-4a7c-8bd9-e82e6700f2b7/resourcegroups/mmlops-RG/workspaces/mosml008", "tags": { "BuildId": "36", "BuildUri": "https://dev.azure.com/mohsmohammad7890561/MLOPS/_build/results?buildId=36" }, "StepTags": { "BuildId": "36", "BuildUri": "https://dev.azure.com/mohsmohammad7890561/MLOPS/_build/results?buildId=36" }, "Properties": {}, "StepProperties": {}, "CreatedBy": { "UserObjectId": "52e7966a-2a24-4e93-883b-50f47ff658e2", "UserTenantId": "39ac8b47-69b1-4db6-b731-b6591aee89d3", "UserName": "3bd719d8-eced-42b7-9a65-5d43bb19f565" }, "PreserveSubGraphs": true, "RootPipelineRunId": "8ed03654-914e-4c2a-b633-3090bdf94669", "EnforceRerun": null, "ContinueRunOnFailedOptionalInput": true, "UserAlias": null, "DefaultCloudPriority": null, "DefaultCompute": null, "DefaultDatastore": null, "IdentityConfig": null, "PipelineTimeoutSeconds": null, "ContinueRunOnStepFailure": false, "EntityStatus": 0, "Id": "8ed03654-914e-4c2a-b633-3090bdf94669", "Etag": "\"4000e293-0000-0300-0000-64ae89470000\"", "CreatedDate": "2023-07-12T11:06:47.1604515Z", "LastModifiedDate": "2023-07-12T11:06:47.1604515Z" }