search

microsoft / MLOpsPython

MLOps using Azure ML Services and Azure DevOps
MIT License
1.21k stars 1.1k forks source link

invoke ml pipeline always failing. The link in the POST Method link gives an error { "Error": { "Code": "UnsupportedApiVersion", "Message": "The HTTP resource that matches the request URI 'https://centralus.aether.ms/api/v1.0/subscriptions/19c1eed4-f0da-4a7c-8bd9-e82e6700f2b7/resourceGroups/mmlops-RG/providers/Microsoft.MachineLearningServices/workspaces/mosml008/PipelineRuns/PipelineSubmit/3ae98478-5ffa-414a-8363-a3e3b3af42db' with API version '1.0' does not support HTTP method 'GET'.", "InnerError": null } } #418

Open codemohsmohammad789 opened 1 year ago

codemohsmohammad789 commented 1 year ago

I am trying to run the train pipeline, I get an error everytime I am unable to figure it out. In the POn ST method link when I run it get an error unsupported AI verson 1.0 error. Please help I have been trying to run the invoke Ml Pipeline from a long time

Job preparation parameters Variables: ACI_DEPLOYMENT_NAME: $[ variablegroups.devopsforai-aml-vg.ACI_DEPLOYMENT_NAME ] AZURE_RM_SVC_CONNECTION: $[ variablegroups.devopsforai-aml-vg.AZURE_RM_SVC_CONNECTION ] BASE_NAME: $[ variablegroups.devopsforai-aml-vg.BASE_NAME ] LOCATION: $[ variablegroups.devopsforai-aml-vg.LOCATION ] RESOURCE_GROUP: $[ variablegroups.devopsforai-aml-vg.RESOURCE_GROUP ] WORKSPACE_NAME: $[ variablegroups.devopsforai-aml-vg.WORKSPACE_NAME ] WORKSPACE_SVC_CONNECTION: $[ variablegroups.devopsforai-aml-vg.WORKSPACE_SVC_CONNECTION ] AMLPIPELINE_ID: Parsing expression: <dependencies.Get_Pipeline_ID.outputs['getpipelineid.AMLPIPELINEID']> Evaluating: dependencies['Get_Pipeline_ID']['outputs']['getpipelineid.AMLPIPELINEID'] Result: '3ae98478-5ffa-414a-8363-a3e3b3af42db' ContinueOnError: False TimeoutInMinutes: 0 CancelTimeoutInMinutes: 5 Expand: MaxConcurrency: 0 MaxConcurrency: 0 ============================================================================== Task : Invoke ML pipeline Description : Invoke Azure Machine Learning published pipelines. Version : 0.1.90 Author : ms-air-aiagility Help URL : https://marketplace.visualstudio.com/items?itemName=ms-air-aiagility.vss-services-azureml&ssr=false#overview ============================================================================== 2023-07-12T11:06:47.6918609Z POST https://centralus.aether.ms/api/v1.0/subscriptions/19c1eed4-f0da-4a7c-8bd9-e82e6700f2b7/resourceGroups/mmlops-RG/providers/Microsoft.MachineLearningServices/workspaces/mosml008/PipelineRuns/PipelineSubmit/3ae98478-5ffa-414a-8363-a3e3b3af42db Request body: { "AzDevOpsCallBackInfo": { "JobId": "77aee9c5-85bc-59ae-fa90-c3f20f9c111e", "HubName": "build", "TaskInstanceId": "7ffef4a3-944d-544a-2764-6fcc510d84c9", "PlanId": "6bebf0d7-465c-4c93-ad8e-49e907470d26", "ProjectId": "0c53d160-630b-4a98-aa55-51747305200b", "PlanUri": "https://dev.azure.com/mohsmohammad7890561/", "Auth": }, "ExperimentName": "mlopspython", "RunSource": "PipelineRun", "ParameterAssignments": { "model_name": "diabetes_regression_model.pkl" }, "tags": { "BuildId": "36", "BuildUri": "https://dev.azure.com/mohsmohammad7890561/MLOPS/_build/results?buildId=36" }, "StepTags": { "BuildId": "36", "BuildUri": "https://dev.azure.com/mohsmohammad7890561/MLOPS/_build/results?buildId=36" } } Response Code: OK Response: { "Description": null, "Status": { "StatusCode": 0, "StatusDetail": null, "CreationTime": "2023-07-12T11:06:47.1409291Z", "EndTime": null }, "GraphId": "98f140f2-05f8-4064-9915-5de767a25c61", "IsSubmitted": false, "HasErrors": false, "HasWarnings": false, "UploadState": 0, "ParameterAssignments": { "model_name": "diabetes_regression_model.pkl", "dataset_version": "latest", "data_file_path": "none", "caller_run_id": "none" }, "DataPathAssignments": {}, "DataSetDefinitionValueAssignments": {}, "AssetOutputSettingsAssignments": {}, "RunHistoryExperimentName": "mlopspython", "DisplayName": null, "PipelineRunId": "8ed03654-914e-4c2a-b633-3090bdf94669", "PipelineId": "3ae98478-5ffa-414a-8363-a3e3b3af42db", "PipelineEndpointId": null, "RunSource": "PipelineRun", "RunType": 0, "TotalRunSteps": 3, "ScheduleId": null, "RunUrl": "https://ml.azure.com/experiments/mlopspython/runs/8ed03654-914e-4c2a-b633-3090bdf94669?tid=39ac8b47-69b1-4db6-b731-b6591aee89d3&wsid=/subscriptions/19c1eed4-f0da-4a7c-8bd9-e82e6700f2b7/resourcegroups/mmlops-RG/workspaces/mosml008", "tags": { "BuildId": "36", "BuildUri": "https://dev.azure.com/mohsmohammad7890561/MLOPS/_build/results?buildId=36" }, "StepTags": { "BuildId": "36", "BuildUri": "https://dev.azure.com/mohsmohammad7890561/MLOPS/_build/results?buildId=36" }, "Properties": {}, "StepProperties": {}, "CreatedBy": { "UserObjectId": "52e7966a-2a24-4e93-883b-50f47ff658e2", "UserTenantId": "39ac8b47-69b1-4db6-b731-b6591aee89d3", "UserName": "3bd719d8-eced-42b7-9a65-5d43bb19f565" }, "PreserveSubGraphs": true, "RootPipelineRunId": "8ed03654-914e-4c2a-b633-3090bdf94669", "EnforceRerun": null, "ContinueRunOnFailedOptionalInput": true, "UserAlias": null, "DefaultCloudPriority": null, "DefaultCompute": null, "DefaultDatastore": null, "IdentityConfig": null, "PipelineTimeoutSeconds": null, "ContinueRunOnStepFailure": false, "EntityStatus": 0, "Id": "8ed03654-914e-4c2a-b633-3090bdf94669", "Etag": "\"4000e293-0000-0300-0000-64ae89470000\"", "CreatedDate": "2023-07-12T11:06:47.1604515Z", "LastModifiedDate": "2023-07-12T11:06:47.1604515Z" }

senar6 commented 2 months ago

Ideally this tasks fails while creating conda environment., and there is no proper error while creating as well. please help to resolve the issue

======Starting Image Build on Serverless Compute====== The run ID for the image build on serverless compute is imgbldrun_725f1f0 Additional logs for the run: https://ml.azure.com/experiments/id/prepare_image/runs/imgbldrun_725f1f0?wsid=/subscriptions/eae29237-49ec-42cc-82e7-dc1125499c88/resourcegroups/rg-sandpox-amrc/workspaces/sandbox-amrc-aml&tid=6f2b364d-3ab6-42e1-a351-e4d69eb92fe9 2024-09-05T15:26:38: Logging into Docker registry: d30f8cdd495e4db28bb266fa51695ba3.azurecr.io 2024-09-05T15:26:38: WARNING! Using --password via the CLI is insecure. Use --password-stdin. 2024-09-05T15:26:38: WARNING! Your password will be stored unencrypted in /root/.docker/config.json. 2024-09-05T15:26:38: Configure a credential helper to remove this warning. See 2024-09-05T15:26:38: https://docs.docker.com/engine/reference/commandline/login/#credentials-store

2024-09-05T15:26:38: Login Succeeded

2024-09-05T15:26:38: Running: ['docker', 'build', '-f', 'azureml-environment-setup/Dockerfile', '.', '-t', 'd30f8cdd495e4db28bb266fa51695ba3.azurecr.io/azureml/azureml_134293382750c1bb119025659155ea48', '-t', 'd30f8cdd495e4db28bb266fa51695ba3.azurecr.io/azureml/azureml_134293382750c1bb119025659155ea48:1'] 2024-09-05T15:26:38: DEPRECATED: The legacy builder is deprecated and will be removed in a future release. 2024-09-05T15:26:38: Install the buildx component to build images with BuildKit: 2024-09-05T15:26:38: https://docs.docker.com/go/buildx/

2024-09-05T15:26:38: Sending build context to Docker daemon 71.17kB 2024-09-05T15:26:38: Step 1/17 : FROM mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04:20210301.v1@sha256:000d6c43f606ceaa67983790ca95c70fd741c364d8c2e3217a11d775b99741df 2024-09-05T15:26:38: mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04:20210301.v1@sha256:000d6c43f606ceaa67983790ca95c70fd741c364d8c2e3217a11d775b99741df: Pulling from azureml/intelmpi2018.3-ubuntu16.04 2024-09-05T15:26:38: 4007a89234b4: Pulling fs layer 2024-09-05T15:26:38: 5dfa26c6b9c9: Pulling fs layer 2024-09-05T15:26:38: 0ba7bf18aa40: Pulling fs layer 2024-09-05T15:26:38: 4c6ec688ebe3: Pulling fs layer 2024-09-05T15:26:38: 574f361512d6: Pulling fs layer 2024-09-05T15:26:38: db4d1e2d7079: Pulling fs layer 2024-09-05T15:26:38: e544ee0f522d: Pulling fs layer 2024-09-05T15:26:38: c655136086be: Pulling fs layer 2024-09-05T15:26:38: 2ec37f44090c: Pulling fs layer 2024-09-05T15:26:38: 5fba3bd4a2c4: Pulling fs layer 2024-09-05T15:26:38: 7e0ea9d0a1ab: Pulling fs layer 2024-09-05T15:26:38: da005f826951: Pulling fs layer 2024-09-05T15:26:38: 4c6ec688ebe3: Waiting 2024-09-05T15:26:38: 574f361512d6: Waiting 2024-09-05T15:26:38: db4d1e2d7079: Waiting 2024-09-05T15:26:38: e544ee0f522d: Waiting 2024-09-05T15:26:38: 7e0ea9d0a1ab: Waiting 2024-09-05T15:26:38: c655136086be: Waiting 2024-09-05T15:26:38: da005f826951: Waiting 2024-09-05T15:26:38: 2ec37f44090c: Waiting 2024-09-05T15:26:38: 5fba3bd4a2c4: Waiting 2024-09-05T15:26:38: 0ba7bf18aa40: Verifying Checksum 2024-09-05T15:26:38: 0ba7bf18aa40: Download complete 2024-09-05T15:26:38: 5dfa26c6b9c9: Download complete 2024-09-05T15:26:38: 4c6ec688ebe3: Verifying Checksum 2024-09-05T15:26:38: 4c6ec688ebe3: Download complete 2024-09-05T15:26:39: db4d1e2d7079: Verifying Checksum 2024-09-05T15:26:39: db4d1e2d7079: Download complete 2024-09-05T15:26:39: 574f361512d6: Verifying Checksum 2024-09-05T15:26:39: 574f361512d6: Download complete 2024-09-05T15:26:40: e544ee0f522d: Verifying Checksum 2024-09-05T15:26:40: e544ee0f522d: Download complete 2024-09-05T15:26:40: 4007a89234b4: Verifying Checksum 2024-09-05T15:26:40: 4007a89234b4: Download complete 2024-09-05T15:26:40: 5fba3bd4a2c4: Verifying Checksum 2024-09-05T15:26:40: 5fba3bd4a2c4: Download complete 2024-09-05T15:26:41: 7e0ea9d0a1ab: Verifying Checksum 2024-09-05T15:26:41: 7e0ea9d0a1ab: Download complete 2024-09-05T15:26:41: da005f826951: Verifying Checksum 2024-09-05T15:26:41: da005f826951: Download complete 2024-09-05T15:26:43: 2ec37f44090c: Verifying Checksum 2024-09-05T15:26:43: 2ec37f44090c: Download complete 2024-09-05T15:26:43: 4007a89234b4: Pull complete 2024-09-05T15:26:44: c655136086be: Verifying Checksum 2024-09-05T15:26:44: c655136086be: Download complete 2024-09-05T15:26:47: 5dfa26c6b9c9: Pull complete 2024-09-05T15:26:47: 0ba7bf18aa40: Pull complete 2024-09-05T15:26:47: 4c6ec688ebe3: Pull complete 2024-09-05T15:26:55: 574f361512d6: Pull complete 2024-09-05T15:26:56: db4d1e2d7079: Pull complete 2024-09-05T15:27:00: e544ee0f522d: Pull complete 2024-09-05T15:27:05: c655136086be: Pull complete 2024-09-05T15:27:11: 2ec37f44090c: Pull complete 2024-09-05T15:27:11: 5fba3bd4a2c4: Pull complete 2024-09-05T15:27:12: 7e0ea9d0a1ab: Pull complete 2024-09-05T15:27:12: da005f826951: Pull complete 2024-09-05T15:27:12: Digest: sha256:000d6c43f606ceaa67983790ca95c70fd741c364d8c2e3217a11d775b99741df 2024-09-05T15:27:12: Status: Downloaded newer image for mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04:20210301.v1@sha256:000d6c43f606ceaa67983790ca95c70fd741c364d8c2e3217a11d775b99741df 2024-09-05T15:27:12: ---> c942df5ba5d0 2024-09-05T15:27:12: Step 2/17 : USER root 2024-09-05T15:27:12: ---> Running in aa672599f5bc 2024-09-05T15:27:13: Removing intermediate container aa672599f5bc 2024-09-05T15:27:13: ---> 16f637ab12ee 2024-09-05T15:27:13: Step 3/17 : RUN mkdir -p $HOME/.cache 2024-09-05T15:27:13: ---> Running in 58efac0e7aec 2024-09-05T15:27:15: Removing intermediate container 58efac0e7aec 2024-09-05T15:27:15: ---> 0b9557f3ff21 2024-09-05T15:27:15: Step 4/17 : WORKDIR / 2024-09-05T15:27:15: ---> Running in 3ee153c2e13f 2024-09-05T15:27:16: Removing intermediate container 3ee153c2e13f 2024-09-05T15:27:16: ---> 856824b7accb 2024-09-05T15:27:16: Step 5/17 : COPY azureml-environment-setup/99brokenproxy /etc/apt/apt.conf.d/ 2024-09-05T15:27:17: ---> be010d5a63a9 2024-09-05T15:27:17: Step 6/17 : RUN if dpkg --compare-versions conda --version | grep -oE '[^ ]+$' lt 4.4.11; then conda install conda==4.4.11; fi 2024-09-05T15:27:17: ---> Running in 8de29dbf8643 2024-09-05T15:27:20: Removing intermediate container 8de29dbf8643 2024-09-05T15:27:20: ---> 736f324cdb84 2024-09-05T15:27:20: Step 7/17 : COPY azureml-environment-setup/mutated_conda_dependencies.yml azureml-environment-setup/mutated_conda_dependencies.yml 2024-09-05T15:27:21: ---> 26d618b1d31e 2024-09-05T15:27:21: Step 8/17 : RUN ldconfig /usr/local/cuda/lib64/stubs && conda env create -p /azureml-envs/azureml_f7d0f75c0d0e79ca21c80a06521c5814 -f azureml-environment-setup/mutated_conda_dependencies.yml && rm -rf "$HOME/.cache/pip" && conda clean -aqy && CONDA_ROOT_DIR=$(conda info --root) && rm -rf "$CONDA_ROOT_DIR/pkgs" && find "$CONDA_ROOT_DIR" -type d -name pycache -exec rm -rf {} + && ldconfig 2024-09-05T15:27:21: ---> Running in 828ca357583d 2024-09-05T15:27:33: Collecting package metadata (repodata.json): ...working... done 2024-09-05T15:27:33: Solving environment: ...working... done

2024-09-05T15:27:33: Downloading and Extracting Packages 2024-09-05T15:27:34: ncurses-6.0 | 781 KB | ########## | 100% 2024-09-05T15:27:34: zlib-1.2.13 | 111 KB | ########## | 100% 2024-09-05T15:27:34: wheel-0.37.1 | 33 KB | ########## | 100% 2024-09-05T15:27:34: sqlite-3.23.1 | 808 KB | ########## | 100% 2024-09-05T15:27:35: python-3.6.2 | 23.6 MB | ########## | 100% 2024-09-05T15:27:35: xz-5.4.6 | 643 KB | ########## | 100% 2024-09-05T15:27:36: tk-8.6.14 | 3.4 MB | ########## | 100% 2024-09-05T15:27:36: readline-7.0 | 848 KB | ########## | 100% 2024-09-05T15:27:36: certifi-2021.5.30 | 139 KB | ########## | 100% 2024-09-05T15:27:36: libgcc-ng-11.2.0 | 5.3 MB | ########## | 100% 2024-09-05T15:27:37: ca-certificates-2024 | 127 KB | ########## | 100% 2024-09-05T15:27:37: libedit-3.1 | 151 KB | ########## | 100% 2024-09-05T15:27:37: libgomp-11.2.0 | 474 KB | ########## | 100% 2024-09-05T15:27:37: libstdcxx-ng-11.2.0 | 4.7 MB | ########## | 100% 2024-09-05T15:27:37: _libgcc_mutex-0.1 | 3 KB | ########## | 100% 2024-09-05T15:27:38: setuptools-58.0.4 | 788 KB | ########## | 100% 2024-09-05T15:27:38: libffi-3.2.1 | 48 KB | ########## | 100% 2024-09-05T15:27:38: openssl-1.0.2u | 2.2 MB | ########## | 100% 2024-09-05T15:27:38: pip-21.2.2 | 1.8 MB | ########## | 100% 2024-09-05T15:27:38: _openmp_mutex-5.1 | 21 KB | ########## | 100% 2024-09-05T15:27:39: Preparing transaction: ...working... done 2024-09-05T15:27:40: Verifying transaction: ...working... done 2024-09-05T15:27:41: Executing transaction: ...working... done