Azure ML Extension: Bug when deploying RTS endpoint using a Model from another Azure Subscription

Describe the bug

Using the Azure ML CLI extension is throwing an error when a real-time endpoint is created using a Model that lives in another Azure subscription. E.g. A model lives in a Model registry in Subscription A. If you provision a real-time endpoint in Subscription B that references the Model in Subscription A it will attempt to provision but eventually throw an error stating that the Resource group could not be found.

The resource group it refers to is the RG in Subscription B where the RTS endpoint would live. When I enable the --debug flag I can see that during processing it gets confused as to what Subscription is should be using, as it eventually combines Subscription A's ID with Subscription B's resource group name in the URI resulting in the 'Resource Group Not Found'.

This used to work when we used an older version of the ML extension (2.25.0) but doesn't work anymore on 2.30.0. We currently have live endpoints that reside in Subscription B but reference Models in our Registry in Subscription A, but new deployments all fail with the above error.

Related command

az ml online-deployment create

Errors

task: [create-deployment] az account set --subscription task: [create-deployment] az ml online-deployment create --endpoint document-classification-prod -f autogenerated_online_deployment.yaml -n prod-v-8-main-58447 --set model=azureml://registries/DataScienceRegistryTest/models/document-classification/versions/8

Uploading s (3.25 MBs): 0%| | 0/3253898 [00:00<?, ?it/s] Uploading s (3.25 MBs): 0%| | 10069/3253898 [00:00<00:32, 98468.03it/s] Uploading s (3.25 MBs): 2%|▏ | 69823/3253898 [00:00<00:08, 387826.80it/s] Uploading s (3.25 MBs): 34%|███▍ | 1106950/3253898 [00:00<00:00, 4810491.73it/s] Uploading s (3.25 MBs): 69%|██████▉ | 2238725/3253898 [00:00<00:00, 5611303.89it/s] Uploading s (3.25 MBs): 86%|████████▌ | 2790292/3253898 [00:00<00:00, 3329528.07it/s] Uploading s (3.25 MBs): 98%|█████████▊| 3194731/3253898 [00:01<00:00, 2666016.27it/s] Uploading s (3.25 MBs): 100%|██████████| 3253898/3253898 [00:01<00:00, 2472003.50it/s]

ERROR: (ResourceGroupNotFound) Resource group 'prod-ml-endpoints' could not be found.

Issue script & Debug output

2024-10-18T15:39:55.8116071Z task: [create-deployment] az account set --subscription 2024-10-18T15:39:56.3208978Z task: [create-deployment] az ml online-deployment create --resource-group "prod-ml-endpoints" --workspace-name --endpoint document-classification-prod -f autogenerated_online_deployment.yaml -n prod-v-8-main-58515 --set model=azureml://registries/DataScienceRegistryTest/models/document-classification/versions/8 --subscription --debug 2024-10-18T15:39:56.5331168Z DEBUG: cli.knack.cli: Command arguments: ['ml', 'online-deployment', 'create', '--resource-group', 'prod-ml-endpoints', '--workspace-name', '', '--endpoint', 'document-classification-prod', '-f', 'autogenerated_online_deployment.yaml', '-n', 'prod-v-8-main-58515', '--set', 'model=azureml://registries/DataScienceRegistryTest/models/document-classification/versions/8', '--subscription', '', '--debug'] 2024-10-18T15:39:56.5332905Z DEBUG: cli.knack.cli: init debug log: 2024-10-18T15:39:56.5333763Z Cannot enable color. 2024-10-18T15:39:56.5334218Z DEBUG: cli.knack.cli: Event: Cli.PreExecute [] 2024-10-18T15:39:56.5338922Z DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f41b7778040>, <function OutputProducer.on_global_arguments at 0x7f41b7726200>, <function CLIQuery.on_global_arguments at 0x7f41b753fce0>] 2024-10-18T15:39:56.5345795Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] 2024-10-18T15:39:59.3090701Z DEBUG: cli.azure.cli.core: ml 2.767 24 161 /opt/az/azcliextensions/ml 2024-10-18T15:39:59.3091159Z DEBUG: cli.azure.cli.core: Total (1) 2.767 24 161
2024-10-18T15:39:59.3091508Z DEBUG: cli.azure.cli.core: Loaded 24 groups, 161 commands. 2024-10-18T15:39:59.3091836Z DEBUG: cli.azure.cli.core: Found a match in the command table. 2024-10-18T15:39:59.3092586Z DEBUG: cli.azure.cli.core: Raw command : ml online-deployment create 2024-10-18T15:39:59.3093038Z DEBUG: cli.azure.cli.core: Command table: ml online-deployment create 2024-10-18T15:39:59.3093458Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f41b6b79da0>] 2024-10-18T15:39:59.3094261Z DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/vsts/work/_temp/.azclitask/commands/2024-10-18.15-39-59.ml_online-deployment_create.2559.log'. 2024-10-18T15:39:59.3095122Z INFO: az_command_data_logger: command args: ml online-deployment create --resource-group {} --workspace-name {} --endpoint {} -f {} -n {} --set {} --subscription {} --debug 2024-10-18T15:39:59.3095663Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7f41b6bb4540>] 2024-10-18T15:39:59.3181670Z INFO: cli.knack.commands: Configured default 'prod-ml-endpoints' for arg resource_group_name 2024-10-18T15:39:59.3182259Z INFO: cli.knack.commands: Configured default '' for arg workspace_name 2024-10-18T15:39:59.3182612Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad [] 2024-10-18T15:39:59.3183256Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7f41b69ca480>, <function register_cache_arguments..add_cache_arguments at 0x7f41b69ca5c0>, <function register_upcoming_breaking_change_info..update_breaking_change_info at 0x7f41b69ca660>] 2024-10-18T15:39:59.3203494Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded [] 2024-10-18T15:39:59.3203860Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [] 2024-10-18T15:39:59.3214269Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f41b77262a0>, <function CLIQuery.handle_query_parameter at 0x7f41b753fd80>, <function register_ids_argument..parse_ids_arguments at 0x7f41b69ca520>] 2024-10-18T15:39:59.3239613Z INFO: az_command_data_logger: extension name: ml 2024-10-18T15:39:59.3239967Z INFO: az_command_data_logger: extension version: 2.30.0 [Redacted] 2024-10-18T15:39:59.3388294Z DEBUG: msal.application: Broker enabled? None 2024-10-18T15:39:59.3425244Z DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.azure.com/.default',), kwargs={} 2024-10-18T15:39:59.3426015Z DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.azure.com/.default',), kwargs={} 2024-10-18T15:39:59.3426436Z DEBUG: msal.application: Cache hit an AT 2024-10-18T15:39:59.3426950Z DEBUG: msal.telemetry: Generate or reuse correlation_id: 1eff4afa-1d1b-44f7-a03f-00fcec2745ca 2024-10-18T15:39:59.3428037Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces/?api-version=2023-04-01-preview' 2024-10-18T15:39:59.3428570Z Request method: 'GET' 2024-10-18T15:39:59.3428812Z Request headers: 2024-10-18T15:39:59.3429175Z 'Accept': 'application/json' 2024-10-18T15:39:59.3429605Z 'x-ms-client-request-id': '39efbce9-8d67-11ef-b286-75264f4eb5e7' 2024-10-18T15:39:59.3430382Z 'User-Agent': 'azureml-cli-v2/2.30.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1073-azure-x86_64-with-glibc2.31) VSTS_260e2ed3-f14f-4b76-a777-9194773130c5_build_632_0' 2024-10-18T15:39:59.3434132Z 'Authorization': 'Bearer 2024-10-18T15:39:59.3434683Z Request body: 2024-10-18T15:39:59.3434923Z This request has no body 2024-10-18T15:39:59.3455922Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 2024-10-18T15:39:59.5128537Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces/?api-version=2023-04-01-preview HTTP/1.1" 200 3310 2024-10-18T15:39:59.5132353Z DEBUG: azure.core.pipeline.policies._universal: Response status: '200' 2024-10-18T15:39:59.5133067Z Response headers: 2024-10-18T15:39:59.5134122Z 'Cache-Control': 'no-cache' 2024-10-18T15:39:59.5134512Z 'Pragma': 'no-cache' 2024-10-18T15:39:59.5134862Z 'Content-Length': '3310' 2024-10-18T15:39:59.5135379Z 'Content-Type': 'application/json; charset=utf-8' 2024-10-18T15:39:59.5135707Z 'Expires': '-1' 2024-10-18T15:39:59.5136018Z 'Vary': 'Accept-Encoding' 2024-10-18T15:39:59.5136426Z 'x-ms-ratelimit-remaining-subscription-global-reads': '3749' 2024-10-18T15:39:59.5136858Z 'x-ms-ratelimit-remaining-subscription-reads': '249' 2024-10-18T15:39:59.5137344Z 'Request-Context': 'appId=cid-v1:0b9f2aab-286f-47f9-a21f-7375db712590' 2024-10-18T15:39:59.5137742Z 'x-ms-response-type': 'standard' 2024-10-18T15:39:59.5138167Z 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' 2024-10-18T15:39:59.5138577Z 'X-Content-Type-Options': 'nosniff' 2024-10-18T15:39:59.5138950Z 'x-aml-cluster': 'vienna-uksouth-02' 2024-10-18T15:39:59.5139287Z 'x-request-time': '0.031' 2024-10-18T15:39:59.5139694Z 'x-ms-request-id': 'e2aba95a-05ad-478c-b42f-35651e86f6a9' 2024-10-18T15:39:59.5140164Z 'x-ms-correlation-request-id': 'e2aba95a-05ad-478c-b42f-35651e86f6a9' 2024-10-18T15:39:59.5140972Z 'x-ms-routing-request-id': 'WESTEUROPE:20241018T153959Z:e2aba95a-05ad-478c-b42f-35651e86f6a9' 2024-10-18T15:39:59.5141383Z 'X-Cache': 'CONFIG_NOCACHE' 2024-10-18T15:39:59.5141910Z 'X-MSEdge-Ref': 'Ref A: 0DDA104351E24DA0A6D48BB94B2087CF Ref B: AMS231020512027 Ref C: 2024-10-18T15:39:59Z' 2024-10-18T15:39:59.5142373Z 'Date': 'Fri, 18 Oct 2024 15:39:58 GMT' 2024-10-18T15:39:59.5142657Z Response content: 2024-10-18T15:39:59.5142873Z { 2024-10-18T15:39:59.5143575Z "id": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces/", 2024-10-18T15:39:59.5144047Z "name": "", 2024-10-18T15:39:59.5144335Z "type": "Microsoft.MachineLearningServices/workspaces", 2024-10-18T15:39:59.5144618Z "location": "uksouth", 2024-10-18T15:39:59.5144862Z "tags": { 2024-10-18T15:39:59.5145098Z "application": "*", 2024-10-18T15:39:59.5145349Z "environment": "prod", 2024-10-18T15:39:59.5145600Z "project": "kyc utility", 2024-10-18T15:39:59.5145847Z "module": "data science" 2024-10-18T15:39:59.5146063Z }, 2024-10-18T15:39:59.5146268Z "etag": null, 2024-10-18T15:39:59.5146490Z "properties": { 2024-10-18T15:39:59.5146861Z "friendlyName": "", 2024-10-18T15:39:59.5147118Z "description": "", 2024-10-18T15:39:59.5147792Z "storageAccount": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/microsoft.storage/storageAccounts/", 2024-10-18T15:39:59.5148564Z "keyVault": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.Keyvault/vaults/-prod-ds-kv26789", 2024-10-18T15:39:59.5149418Z "applicationInsights": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.insights/components/-prod-ds-appinsights26789", 2024-10-18T15:39:59.5149825Z "hbiWorkspace": false, 2024-10-18T15:39:59.5150214Z "tenantId": "", 2024-10-18T15:39:59.5150586Z "imageBuildCompute": "cpu-cluster", 2024-10-18T15:39:59.5150857Z "provisioningState": "Succeeded", 2024-10-18T15:39:59.5151107Z "managedNetwork": { 2024-10-18T15:39:59.5151354Z "changeableIsolationModes": [ 2024-10-18T15:39:59.5151627Z "AllowInternetOutbound", 2024-10-18T15:39:59.5151888Z "AllowOnlyApprovedOutbound" 2024-10-18T15:39:59.5152110Z ], 2024-10-18T15:39:59.5152339Z "isolationMode": "Disabled" 2024-10-18T15:39:59.5152563Z }, 2024-10-18T15:39:59.5152800Z "allowRoleAssignmentOnRG": true, 2024-10-18T15:39:59.5153052Z "v1LegacyMode": false, 2024-10-18T15:39:59.5153301Z "softDeleteEnabled": false, 2024-10-18T15:39:59.5154166Z "containerRegistry": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.ContainerRegistry/registries/d896b339ae604759be496aad9479a948", 2024-10-18T15:39:59.5155025Z "creationTime": "2023-08-11T13:26:03.8187396Z", 2024-10-18T15:39:59.5155306Z "notebookInfo": { 2024-10-18T15:39:59.5155583Z "resourceId": "752018ff0ef9442e808bc89ca36833c8", 2024-10-18T15:39:59.5156116Z "fqdn": "ml-*-prod-d-uksouth-d896b339-ae60-4759-be49-6aad9479a948.uksouth.notebooks.azure.net", 2024-10-18T15:39:59.5156461Z "isPrivateLinkEnabled": false, 2024-10-18T15:39:59.5156727Z "notebookPreparationError": null 2024-10-18T15:39:59.5156958Z }, 2024-10-18T15:39:59.5157202Z "storageHnsEnabled": false, 2024-10-18T15:39:59.5157598Z "workspaceId": "d896b339-ae60-4759-be49-6aad9479a948", 2024-10-18T15:39:59.5157893Z "linkedModelInventoryArmId": null, 2024-10-18T15:39:59.5158151Z "privateLinkCount": 0, 2024-10-18T15:39:59.5158421Z "publicNetworkAccess": "Enabled", 2024-10-18T15:39:59.5158725Z "discoveryUrl": "https://uksouth.api.azureml.ms/discovery", 2024-10-18T15:39:59.5159599Z "mlFlowTrackingUri": "azureml://uksouth.api.azureml.ms/mlflow/v1.0/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces/", 2024-10-18T15:39:59.5160377Z "sdkTelemetryAppInsightsKey": "2cd6c47a-322c-4d3a-a2e8-645d0fade7df", 2024-10-18T15:39:59.5160683Z "enableDataIsolation": false, 2024-10-18T15:39:59.5160952Z "systemDatastoresAuthMode": null, 2024-10-18T15:39:59.5161232Z "enableServiceSideCMKEncryption": false 2024-10-18T15:39:59.5161475Z }, 2024-10-18T15:39:59.5161683Z "identity": { 2024-10-18T15:39:59.5161923Z "type": "SystemAssigned", 2024-10-18T15:39:59.5162321Z "principalId": "f3f18dbb-a247-4947-b549-a4844ad9cee5", 2024-10-18T15:39:59.5162768Z "tenantId": "" 2024-10-18T15:39:59.5163022Z }, 2024-10-18T15:39:59.5163237Z "kind": "Default", 2024-10-18T15:39:59.5163452Z "sku": { 2024-10-18T15:39:59.5163669Z "name": "Basic", 2024-10-18T15:39:59.5163896Z "tier": "Basic" 2024-10-18T15:39:59.5164107Z }, 2024-10-18T15:39:59.5164316Z "systemData": { 2024-10-18T15:39:59.5164690Z "createdAt": "2023-08-11T13:26:03.8187396Z", 2024-10-18T15:39:59.5165011Z "createdBy": "@.com", 2024-10-18T15:39:59.5165308Z "createdByType": "User", 2024-10-18T15:39:59.5165700Z "lastModifiedAt": "2023-10-13T14:55:42.5201466Z", 2024-10-18T15:39:59.5166015Z "lastModifiedBy": "@.com", 2024-10-18T15:39:59.5166293Z "lastModifiedByType": "User" 2024-10-18T15:39:59.5166517Z } 2024-10-18T15:39:59.5166710Z } 2024-10-18T15:39:59.5197967Z DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.azure.com/.default',), kwargs={} 2024-10-18T15:39:59.5198842Z DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.azure.com/.default',), kwargs={} 2024-10-18T15:39:59.5199266Z DEBUG: msal.application: Cache hit an AT 2024-10-18T15:39:59.5199769Z DEBUG: msal.telemetry: Generate or reuse correlation_id: 7925f6cb-3044-44b6-8b2b-cdd8d6241b01 2024-10-18T15:39:59.5200455Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://eastus.api.azureml.ms/registrymanagement/v1.0/registries/DataScienceRegistryTest/discovery' 2024-10-18T15:39:59.5200879Z Request method: 'GET' 2024-10-18T15:39:59.5201120Z Request headers: 2024-10-18T15:39:59.5201462Z 'Accept': 'application/json, text/json' 2024-10-18T15:39:59.5201901Z 'x-ms-client-request-id': '39efbce9-8d67-11ef-b286-75264f4eb5e7' 2024-10-18T15:39:59.5202688Z 'User-Agent': 'azureml-cli-v2/2.30.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1073-azure-x86_64-with-glibc2.31) VSTS_260e2ed3-f14f-4b76-a777-9194773130c5_build_632_0' 2024-10-18T15:39:59.5206793Z 'Authorization': 'Bearer 2024-10-18T15:39:59.5207045Z Request body: 2024-10-18T15:39:59.5207284Z This request has no body 2024-10-18T15:39:59.5226043Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): eastus.api.azureml.ms:443 2024-10-18T15:40:00.5040786Z DEBUG: urllib3.connectionpool: https://eastus.api.azureml.ms:443 "GET /registrymanagement/v1.0/registries/DataScienceRegistryTest/discovery HTTP/1.1" 200 None 2024-10-18T15:40:00.5058106Z DEBUG: azure.core.pipeline.policies._universal: Response status: '200' 2024-10-18T15:40:00.5058563Z Response headers: 2024-10-18T15:40:00.5059030Z 'Date': 'Fri, 18 Oct 2024 15:40:00 GMT' 2024-10-18T15:40:00.5059567Z 'Content-Type': 'application/json; charset=utf-8' 2024-10-18T15:40:00.5060014Z 'Transfer-Encoding': 'chunked' 2024-10-18T15:40:00.5060400Z 'Connection': 'keep-alive' 2024-10-18T15:40:00.5060781Z 'Vary': 'Accept-Encoding' 2024-10-18T15:40:00.5061304Z 'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d' 2024-10-18T15:40:00.5061792Z 'x-ms-response-type': 'standard' 2024-10-18T15:40:00.5062324Z 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload' 2024-10-18T15:40:00.5062801Z 'X-Content-Type-Options': 'nosniff' 2024-10-18T15:40:00.5063605Z 'x-aml-cluster': 'vienna-eastus-02' 2024-10-18T15:40:00.5064000Z 'x-request-time': '0.619' 2024-10-18T15:40:00.5064406Z 'Content-Encoding': 'gzip' 2024-10-18T15:40:00.5064696Z Response content: 2024-10-18T15:40:00.5064953Z { 2024-10-18T15:40:00.5065388Z "registryId": "6aba4a2a-47d1-4326-9764-a244ae2dfb59", 2024-10-18T15:40:00.5065752Z "registryName": "DataScienceRegistryTest", 2024-10-18T15:40:00.5066231Z "tenantId": "", 2024-10-18T15:40:00.5066564Z "primaryRegion": "uksouth", 2024-10-18T15:40:00.5066837Z "regions": [ 2024-10-18T15:40:00.5067090Z "uksouth" 2024-10-18T15:40:00.5067346Z ], 2024-10-18T15:40:00.5067785Z "subscriptionId": "", 2024-10-18T15:40:00.5068250Z "resourceGroup": "generic-resource-group", 2024-10-18T15:40:00.5068560Z "workspaceName": null, 2024-10-18T15:40:00.5069083Z "primaryRegionResourceProviderUri": "https://cert-uksouth.experiments.azureml.net/", 2024-10-18T15:40:00.5069447Z "registryFqdns": { 2024-10-18T15:40:00.5069714Z "uksouth": { 2024-10-18T15:40:00.5070016Z "uri": "https://uksouth.api.azureml.ms" 2024-10-18T15:40:00.5070292Z } 2024-10-18T15:40:00.5070521Z }, 2024-10-18T15:40:00.5070821Z "intellectualPropertyPublisher": null, 2024-10-18T15:40:00.5071138Z "isTradeRestricted": false 2024-10-18T15:40:00.5071398Z } 2024-10-18T15:40:00.5204161Z DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.azure.com/.default',), kwargs={} 2024-10-18T15:40:00.5207920Z DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.azure.com/.default',), kwargs={} 2024-10-18T15:40:00.5209761Z DEBUG: msal.application: Cache hit an AT 2024-10-18T15:40:00.5216245Z DEBUG: msal.telemetry: Generate or reuse correlation_id: 257709cb-218f-4989-ab70-70cda9f15065 2024-10-18T15:40:00.5217737Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//onlineEndpoints/document-classification-prod/deployments/prod-v-8-main-58515?api-version=2023-04-01-preview' 2024-10-18T15:40:00.5218482Z Request method: 'GET' 2024-10-18T15:40:00.5218784Z Request headers: 2024-10-18T15:40:00.5219159Z 'Accept': 'application/json' 2024-10-18T15:40:00.5219663Z 'x-ms-client-request-id': '39efbce9-8d67-11ef-b286-75264f4eb5e7' 2024-10-18T15:40:00.5220622Z 'User-Agent': 'azureml-cli-v2/2.30.0 azure-ai-ml/1.20.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1073-azure-x86_64-with-glibc2.31) VSTS_260e2ed3-f14f-4b76-a777-9194773130c5_build_632_0' 2024-10-18T15:40:00.5225497Z 'Authorization': 'Bearer 2024-10-18T15:40:00.5225788Z Request body: 2024-10-18T15:40:00.5226076Z This request has no body 2024-10-18T15:40:00.5239575Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 2024-10-18T15:40:00.7094637Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//onlineEndpoints/document-classification-prod/deployments/prod-v-8-main-58515?api-version=2023-04-01-preview HTTP/1.1" 404 345 2024-10-18T15:40:00.7095707Z DEBUG: azure.core.pipeline.policies._universal: Response status: '404' 2024-10-18T15:40:00.7096055Z Response headers: 2024-10-18T15:40:00.7096441Z 'Cache-Control': 'no-cache' 2024-10-18T15:40:00.7096810Z 'Pragma': 'no-cache' 2024-10-18T15:40:00.7097198Z 'Content-Length': '345' 2024-10-18T15:40:00.7097644Z 'Content-Type': 'application/json; charset=utf-8' 2024-10-18T15:40:00.7098053Z 'Expires': '-1' 2024-10-18T15:40:00.7098430Z 'x-ms-failure-cause': 'gateway' 2024-10-18T15:40:00.7098925Z 'x-ms-request-id': '7641333b-6af3-4e11-8320-31da9b5e84bc' 2024-10-18T15:40:00.7099475Z 'x-ms-correlation-request-id': '7641333b-6af3-4e11-8320-31da9b5e84bc' 2024-10-18T15:40:00.7100467Z 'x-ms-routing-request-id': 'WESTEUROPE:20241018T154000Z:7641333b-6af3-4e11-8320-31da9b5e84bc' 2024-10-18T15:40:00.7101130Z 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' 2024-10-18T15:40:00.7101613Z 'X-Content-Type-Options': 'nosniff' 2024-10-18T15:40:00.7102018Z 'X-Cache': 'CONFIG_NOCACHE' 2024-10-18T15:40:00.7102634Z 'X-MSEdge-Ref': 'Ref A: F3CDB0E25AC945449F7AB4983C1C282E Ref B: AMS231032608035 Ref C: 2024-10-18T15:40:00Z' 2024-10-18T15:40:00.7103173Z 'Date': 'Fri, 18 Oct 2024 15:39:59 GMT' 2024-10-18T15:40:00.7103487Z Response content: 2024-10-18T15:40:00.7104690Z {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.MachineLearningServices/workspaces//onlineEndpoints/document-classification-prod/deployments/prod-v-8-main-58515' under resource group '' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}} 2024-10-18T15:40:00.7110630Z Check: endpoint document-classification-prod exists 2024-10-18T15:40:00.7120433Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//onlineEndpoints/document-classification-prod?api-version=2023-04-01-preview' 2024-10-18T15:40:00.7121162Z Request method: 'GET' 2024-10-18T15:40:00.7121452Z Request headers: 2024-10-18T15:40:00.7121826Z 'Accept': 'application/json' 2024-10-18T15:40:00.7122329Z 'x-ms-client-request-id': '39efbce9-8d67-11ef-b286-75264f4eb5e7' 2024-10-18T15:40:00.7123289Z 'User-Agent': 'azureml-cli-v2/2.30.0 azure-ai-ml/1.20.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1073-azure-x86_64-with-glibc2.31) VSTS_260e2ed3-f14f-4b76-a777-9194773130c5_build_632_0' 2024-10-18T15:40:00.7127767Z 'Authorization': 'Bearer 2024-10-18T15:40:00.7128080Z Request body: 2024-10-18T15:40:00.7128367Z This request has no body 2024-10-18T15:40:00.9976254Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//onlineEndpoints/document-classification-prod?api-version=2023-04-01-preview HTTP/1.1" 200 1831 2024-10-18T15:40:00.9984149Z DEBUG: azure.core.pipeline.policies._universal: Response status: '200' 2024-10-18T15:40:00.9996196Z Response content: 2024-10-18T15:40:00.9996419Z { 2024-10-18T15:40:00.9997697Z "id": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//onlineEndpoints/document-classification-prod", 2024-10-18T15:40:00.9998219Z "name": "document-classification-prod", 2024-10-18T15:40:00.9998542Z "type": "Microsoft.MachineLearningServices/workspaces/onlineEndpoints", 2024-10-18T15:40:00.9998830Z "properties": { 2024-10-18T15:40:00.9999120Z "description": "Document classification endpoint", 2024-10-18T15:40:00.9999389Z "properties": { 2024-10-18T15:40:01.0000227Z "azureml.onlineendpointid": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/microsoft.machinelearningservices/workspaces//onlineendpoints/document-classification-prod", 2024-10-18T15:40:01.0001383Z "AzureAsyncOperationUri": "https://management.azure.com/subscriptions//providers/Microsoft.MachineLearningServices/locations/uksouth/mfeOperationsStatus/oeidp:d896b339-ae60-4759-be49-6aad9479a948:ac297511-a021-4ae2-bf1a-7ba06f80d282?api-version=2023-04-01-preview" 2024-10-18T15:40:01.0001944Z }, 2024-10-18T15:40:01.0002387Z "scoringUri": ".inference.ml.azure.com/score", 2024-10-18T15:40:01.0002931Z "swaggerUri": ".inference.ml.azure.com/swagger.json", 2024-10-18T15:40:01.0003252Z "authMode": "Key", 2024-10-18T15:40:01.0003512Z "provisioningState": "Succeeded", 2024-10-18T15:40:01.0003762Z "compute": null, 2024-10-18T15:40:01.0004020Z "publicNetworkAccess": "Enabled", 2024-10-18T15:40:01.0004260Z "traffic": { 2024-10-18T15:40:01.0004593Z "prod-v-2-main-50078": 100 2024-10-18T15:40:01.0004815Z }, 2024-10-18T15:40:01.0005033Z "mirrorTraffic": {} 2024-10-18T15:40:01.0005245Z }, 2024-10-18T15:40:01.0005461Z "systemData": { 2024-10-18T15:40:01.0006843Z "createdAt": "2024-06-14T13:36:00.627183+00:00", 2024-10-18T15:40:01.0007294Z "createdBy": "7397dc71-2601-4b99-b2e8-2cfdb54c999e", 2024-10-18T15:40:01.0007723Z "lastModifiedAt": "2024-06-14T13:36:00.627183+00:00" 2024-10-18T15:40:01.0007987Z }, 2024-10-18T15:40:01.0008187Z "tags": {}, 2024-10-18T15:40:01.0008438Z "location": "uksouth", 2024-10-18T15:40:01.0008671Z "kind": "Managed", 2024-10-18T15:40:01.0008896Z "identity": { 2024-10-18T15:40:01.0009132Z "type": "SystemAssigned", 2024-10-18T15:40:01.0009525Z "principalId": "186e2b15-7370-4155-af76-fc5fc9a479a5", 2024-10-18T15:40:01.0009940Z "tenantId": "" 2024-10-18T15:40:01.0010395Z } 2024-10-18T15:40:01.0010586Z } 2024-10-18T15:40:01.1093299Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//codes/98583559-9974-4d09-9d95-ef35a7bd57f2/versions?api-version=2023-04-01&hash=8a26422e1f56065ab6e90b61a753828aac8c065d20bd9342fc2767b9896efb49&hashVersion=202208' 2024-10-18T15:40:01.1094000Z Request method: 'GET' 2024-10-18T15:40:01.1094240Z Request headers: 2024-10-18T15:40:01.1094559Z 'Accept': 'application/json' 2024-10-18T15:40:01.1094984Z 'x-ms-client-request-id': '39efbce9-8d67-11ef-b286-75264f4eb5e7' 2024-10-18T15:40:01.1096137Z 'User-Agent': 'azureml-cli-v2/2.30.0 azure-ai-ml/1.20.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1073-azure-x86_64-with-glibc2.31) VSTS_260e2ed3-f14f-4b76-a777-9194773130c5_build_632_0' 2024-10-18T15:40:01.1099930Z 'Authorization': 'Bearer 2024-10-18T15:40:01.1100174Z Request body: 2024-10-18T15:40:01.1100412Z This request has no body 2024-10-18T15:40:01.1123165Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 2024-10-18T15:40:02.2326574Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//codes/98583559-9974-4d09-9d95-ef35a7bd57f2/versions?api-version=2023-04-01&hash=8a26422e1f56065ab6e90b61a753828aac8c065d20bd9342fc2767b9896efb49&hashVersion=202208 HTTP/1.1" 200 17 2024-10-18T15:40:02.2335981Z DEBUG: azure.core.pipeline.policies._universal: Response status: '200' 2024-10-18T15:40:02.2336386Z Response headers: 2024-10-18T15:40:02.2336839Z 'Cache-Control': 'no-cache' 2024-10-18T15:40:02.2337213Z 'Pragma': 'no-cache' 2024-10-18T15:40:02.2337580Z 'Content-Length': '17' 2024-10-18T15:40:02.2338042Z 'Content-Type': 'application/json; charset=utf-8' 2024-10-18T15:40:02.2338424Z 'Expires': '-1' 2024-10-18T15:40:02.2338871Z 'x-ms-ratelimit-remaining-subscription-reads': '249' 2024-10-18T15:40:02.2339441Z 'Request-Context': 'appId=cid-v1:0b9f2aab-286f-47f9-a21f-7375db712590' 2024-10-18T15:40:02.2339912Z 'x-ms-response-type': 'standard' 2024-10-18T15:40:02.2340414Z 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' 2024-10-18T15:40:02.2340886Z 'X-Content-Type-Options': 'nosniff' 2024-10-18T15:40:02.2341309Z 'x-aml-cluster': 'vienna-uksouth-01' 2024-10-18T15:40:02.2341721Z 'x-request-time': '0.176' 2024-10-18T15:40:02.2342198Z 'x-ms-ratelimit-remaining-subscription-global-reads': '3749' 2024-10-18T15:40:02.2342731Z 'x-ms-request-id': 'c3d61d34-4817-44e2-abeb-1349d9c1a492' 2024-10-18T15:40:02.2343276Z 'x-ms-correlation-request-id': 'c3d61d34-4817-44e2-abeb-1349d9c1a492' 2024-10-18T15:40:02.2343923Z 'x-ms-routing-request-id': 'WESTEUROPE:20241018T154002Z:c3d61d34-4817-44e2-abeb-1349d9c1a492' 2024-10-18T15:40:02.2344407Z 'X-Cache': 'CONFIG_NOCACHE' 2024-10-18T15:40:02.2345022Z 'X-MSEdge-Ref': 'Ref A: 04A993C5A04E496DA1FCB696D27D3991 Ref B: AMS231032609035 Ref C: 2024-10-18T15:40:01Z' 2024-10-18T15:40:02.2345562Z 'Date': 'Fri, 18 Oct 2024 15:40:01 GMT' 2024-10-18T15:40:02.2345893Z Response content: 2024-10-18T15:40:02.2346147Z { 2024-10-18T15:40:02.2346385Z "value": [] 2024-10-18T15:40:02.2346623Z } 2024-10-18T15:40:02.2360695Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//codes/98583559-9974-4d09-9d95-ef35a7bd57f2/versions/1/startPendingUpload?api-version=2023-04-01' 2024-10-18T15:40:02.2361544Z Request method: 'POST' 2024-10-18T15:40:02.2362214Z Request headers: 2024-10-18T15:40:02.2362616Z 'Content-Type': 'application/json' 2024-10-18T15:40:02.2363024Z 'Content-Length': '47' 2024-10-18T15:40:02.2365779Z 'Accept': 'application/json' 2024-10-18T15:40:02.2366829Z 'x-ms-client-request-id': '39efbce9-8d67-11ef-b286-75264f4eb5e7' 2024-10-18T15:40:02.2368541Z 'User-Agent': 'azureml-cli-v2/2.30.0 azure-ai-ml/1.20.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1073-azure-x86_64-with-glibc2.31) VSTS_260e2ed3-f14f-4b76-a777-9194773130c5_build_632_0' 2024-10-18T15:40:02.2373171Z 'Authorization': 'Bearer 2024-10-18T15:40:02.2373478Z Request body: 2024-10-18T15:40:02.2373801Z {"pendingUploadType": "TemporaryBlobReference"} 2024-10-18T15:40:03.5882150Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "POST /subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//codes/98583559-9974-4d09-9d95-ef35a7bd57f2/versions/1/startPendingUpload?api-version=2023-04-01 HTTP/1.1" 200 952 2024-10-18T15:40:03.5890511Z DEBUG: azure.core.pipeline.policies._universal: Response status: '200' 2024-10-18T15:40:03.5890972Z Response headers: [Redacted] 2024-10-18T15:40:03.5901347Z Response content: 2024-10-18T15:40:03.5901600Z { 2024-10-18T15:40:03.5901876Z "blobReferenceForConsumption": { 2024-10-18T15:40:03.5902694Z "blobUri": "https://***.blob.core.windows.net:443/d896b339-ae60-4759-be49-6aad9479a948-k7zq66rie9alphxv6fjtjebmgz", 2024-10-18T15:40:03.5903650Z "storageAccountArmId": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.Storage/storageAccounts/", 2024-10-18T15:40:03.5904079Z "credential": { 2024-10-18T15:40:03.5904365Z "credentialType": "SAS", 2024-10-18T15:40:03.5905889Z "sasUri": "https://.blob.core.windows.net/d896b339-ae60-4759-be49-6aad9479a948-k7zq66rie9alphxv6fjtjebmgz?skoid=f3f18dbb-a247-4947-b549-a4844ad9cee5&sktid=&skt=2024-10-18T12%3A25%3A00Z&ske=2024-10-19T04%3A35%3A00Z&sks=b&skv=2019-07-07&sv=2021-10-04&st=2024-10-18T15%3A30%3A03Z&se=2024-10-18T23%3A40%3A03Z&sr=c&sp=rcwl&sig=96e7SabWjC7pbcAAXpJSYT2xwFRDgg%2FdpFbTEPlXJjc%3D" 2024-10-18T15:40:03.5906936Z } 2024-10-18T15:40:03.5907170Z }, 2024-10-18T15:40:03.5907626Z "pendingUploadId": "1438792b-eac4-46b5-a71b-8c90165bafa1", 2024-10-18T15:40:03.5907974Z "pendingUploadType": "None" 2024-10-18T15:40:03.5908251Z } 2024-10-18T15:40:03.6915406Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): .blob.core.windows.net:443 [REDACTED -- Bunch of blob uploads. Everything succeeds] [REDACTED -- Bunch of blob uploads. Everything succeeds] [REDACTED -- Bunch of blob uploads. Everything succeeds] [REDACTED -- Bunch of blob uploads. Everything succeeds] [REDACTED -- Bunch of blob uploads. Everything succeeds] 2024-10-18T15:40:05.3588900Z Uploading s (3.26 MBs): 100%|██████████| 3255200/3255200 [00:01<00:00, 2520841.94it/s] 2024-10-18T15:40:05.3597083Z 2024-10-18T15:40:05.3597549Z 2024-10-18T15:40:05.3775512Z DEBUG: urllib3.connectionpool: https://.blob.core.windows.net:443 "PUT /d896b339-ae60-4759-be49-6aad9479a948-k7zq66rie9alphxv6fjtjebmgz/s/.git/FETCH_HEAD?comp=metadata&skoid=f3f18dbb-a247-4947-b549-a4844ad9cee5&sktid=&skt=2024-10-18T12%3A25%3A00Z&ske=2024-10-19T04%3A35%3A00Z&sks=b&skv=2019-07-07&sv=2021-10-04&st=2024-10-18T15%3A30%3A03Z&se=2024-10-18T23%3A40%3A03Z&sr=c&sp=rcwl&sig=96e7SabWjC7pbcAAXpJSYT2xwFRDgg%2FdpFbTEPlXJjc%3D HTTP/1.1" 200 0 2024-10-18T15:40:05.3799031Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//codes/98583559-9974-4d09-9d95-ef35a7bd57f2/versions/1?api-version=2023-04-01' 2024-10-18T15:40:05.3799858Z Request method: 'PUT' 2024-10-18T15:40:05.3800145Z Request headers: 2024-10-18T15:40:05.3800537Z 'Content-Type': 'application/json' 2024-10-18T15:40:05.3800927Z 'Content-Length': '319' 2024-10-18T15:40:05.3801315Z 'Accept': 'application/json' 2024-10-18T15:40:05.3801812Z 'x-ms-client-request-id': '39efbce9-8d67-11ef-b286-75264f4eb5e7' 2024-10-18T15:40:05.3802784Z 'User-Agent': 'azureml-cli-v2/2.30.0 azure-ai-ml/1.20.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1073-azure-x86_64-with-glibc2.31) VSTS_260e2ed3-f14f-4b76-a777-9194773130c5_build_632_0' 2024-10-18T15:40:05.3807259Z 'Authorization': 'Bearer 2024-10-18T15:40:05.3807551Z Request body: 2024-10-18T15:40:05.3808771Z {"properties": {"properties": {"hash_sha256": "8a26422e1f56065ab6e90b61a753828aac8c065d20bd9342fc2767b9896efb49", "hash_version": "202208"}, "isAnonymous": true, "isArchived": false, "codeUri": "https://***.blob.core.windows.net:443/d896b339-ae60-4759-be49-6aad9479a948-k7zq66rie9alphxv6fjtjebmgz/s"}} 2024-10-18T15:40:06.9890065Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//codes/98583559-9974-4d09-9d95-ef35a7bd57f2/versions/1?api-version=2023-04-01 HTTP/1.1" 201 1100 2024-10-18T15:40:06.9891126Z DEBUG: azure.core.pipeline.policies._universal: Response status: '201' 2024-10-18T15:40:06.9891536Z Response headers: [REDACTED] 2024-10-18T15:40:06.9893567Z 'Expires': '-1' 2024-10-18T15:40:06.9894672Z 'Location': 'https://management.azure.com/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//codes/98583559-9974-4d09-9d95-ef35a7bd57f2/versions/1?api-version=2023-04-01' [REDACTED] 2024-10-18T15:40:06.9902491Z Response content: 2024-10-18T15:40:06.9902749Z { 2024-10-18T15:40:06.9903677Z "id": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//codes/98583559-9974-4d09-9d95-ef35a7bd57f2/versions/1", 2024-10-18T15:40:06.9904151Z "name": "1", 2024-10-18T15:40:06.9904848Z "type": "Microsoft.MachineLearningServices/workspaces/codes/versions", 2024-10-18T15:40:06.9905187Z "properties": { 2024-10-18T15:40:06.9905468Z "description": null, 2024-10-18T15:40:06.9905738Z "tags": {}, 2024-10-18T15:40:06.9906006Z "properties": { 2024-10-18T15:40:06.9906379Z "hash_sha256": "8a26422e1f56065ab6e90b61a753828aac8c065d20bd9342fc2767b9896efb49", 2024-10-18T15:40:06.9906748Z "hash_version": "202208" 2024-10-18T15:40:06.9907022Z }, 2024-10-18T15:40:06.9907285Z "isArchived": false, 2024-10-18T15:40:06.9907572Z "isAnonymous": false, 2024-10-18T15:40:06.9908304Z "codeUri": "https://.blob.core.windows.net:443/d896b339-ae60-4759-be49-6aad9479a948-k7zq66rie9alphxv6fjtjebmgz/s", 2024-10-18T15:40:06.9908745Z "provisioningState": "Succeeded" 2024-10-18T15:40:06.9915585Z }, 2024-10-18T15:40:06.9915935Z "systemData": { 2024-10-18T15:40:06.9916504Z "createdAt": "2024-10-18T15:40:06.6798022+00:00", 2024-10-18T15:40:06.9917059Z "createdBy": "", 2024-10-18T15:40:06.9917373Z "createdByType": "Application", 2024-10-18T15:40:06.9917863Z "lastModifiedAt": "2024-10-18T15:40:06.6798022+00:00", 2024-10-18T15:40:06.9918403Z "lastModifiedBy": "", 2024-10-18T15:40:06.9918716Z "lastModifiedByType": "Application" 2024-10-18T15:40:06.9919191Z } 2024-10-18T15:40:06.9919417Z } 2024-10-18T15:40:06.9920745Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//environments/dc-model-prod/versions?api-version=2023-04-01-preview&$orderBy=createdtime%20desc&$top=1' 2024-10-18T15:40:06.9921448Z Request method: 'GET' 2024-10-18T15:40:06.9921735Z Request headers: 2024-10-18T15:40:06.9922130Z 'Accept': 'application/json' 2024-10-18T15:40:06.9922628Z 'x-ms-client-request-id': '39efbce9-8d67-11ef-b286-75264f4eb5e7' 2024-10-18T15:40:06.9923609Z 'User-Agent': 'azureml-cli-v2/2.30.0 azure-ai-ml/1.20.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1073-azure-x86_64-with-glibc2.31) VSTS_260e2ed3-f14f-4b76-a777-9194773130c5_build_632_0' 2024-10-18T15:40:06.9928081Z 'Authorization': 'Bearer 2024-10-18T15:40:06.9928375Z Request body: 2024-10-18T15:40:06.9928655Z This request has no body 2024-10-18T15:40:07.5385964Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//environments/dc-model-prod/versions?api-version=2023-04-01-preview&$orderBy=createdtime%20desc&$top=1 HTTP/1.1" 200 6204 2024-10-18T15:40:07.5386938Z DEBUG: azure.core.pipeline.policies._universal: Response status: '200' 2024-10-18T15:40:07.5387213Z Response headers: [REDACTED] 2024-10-18T15:40:07.5395628Z Response content: 2024-10-18T15:40:07.5395828Z { 2024-10-18T15:40:07.5396016Z "value": [ 2024-10-18T15:40:07.5396207Z { 2024-10-18T15:40:07.5396916Z "id": "/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces//environments/dc-model-prod/versions/17", 2024-10-18T15:40:07.5397284Z "name": "17", 2024-10-18T15:40:07.5397619Z "type": "Microsoft.MachineLearningServices/workspaces/environments/versions", 2024-10-18T15:40:07.5397892Z "properties": { 2024-10-18T15:40:07.5398116Z "description": null, 2024-10-18T15:40:07.5398330Z "tags": {}, 2024-10-18T15:40:07.5398540Z "properties": { 2024-10-18T15:40:07.5398941Z "azureml.labels": "latest" 2024-10-18T15:40:07.5399201Z }, 2024-10-18T15:40:07.5399428Z "isArchived": false, 2024-10-18T15:40:07.5399805Z "isAnonymous": false, 2024-10-18T15:40:07.5400069Z "autoDeleteSetting": null, 2024-10-18T15:40:07.5400337Z "environmentType": "UserCreated", 2024-10-18T15:40:07.5400827Z "image": "mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference:latest", 2024-10-18T15:40:07.5412407Z "condaFile": "{\n \"channels\": [\n \"conda-forge\"\n ],\n \"dependencies\": [\n \"_libgcc_mutex=0.1=conda_forge\",\n \"_openmp_mutex=4.5=2_gnu\",\n \"annotated-types=0.7.0=pyhd8ed1ab_0\",\n \"brotli-python=1.1.0=py310hc6cd4ac_1\",\n \"bzip2=1.0.8=hd590300_5\",\n \"ca-certificates=2024.6.2=hbcca054_0\",\n \"certifi=2024.6.2=pyhd8ed1ab_0\",\n \"charset-normalizer=3.3.2=pyhd8ed1ab_0\",\n \"click=8.1.7=unix_pyh707e725_0\",\n \"cloudpickle=3.0.0=pyhd8ed1ab_0\",\n \"colorama=0.4.6=pyhd8ed1ab_0\",\n \"entrypoints=0.4=pyhd8ed1ab_0\",\n \"exceptiongroup=1.2.0=pyhd8ed1ab_2\",\n \"gitdb=4.0.11=pyhd8ed1ab_0\",\n \"gitpython=3.1.43=pyhd8ed1ab_0\",\n \"idna=3.7=pyhd8ed1ab_0\",\n \"importlib-metadata=7.1.0=pyha770c72_0\",\n \"iniconfig=2.0.0=pyhd8ed1ab_0\",\n \"joblib=1.4.2=pyhd8ed1ab_0\",\n \"ld_impl_linux-64=2.40=hf3520f5_3\",\n \"libabseil=20240116.2=cxx17_h59595ed_0\",\n \"libblas=3.9.0=22_linux64_openblas\",\n \"libcblas=3.9.0=22_linux64_openblas\",\n \"libffi=3.4.2=h7f98852_5\",\n \"libgcc-ng=13.2.0=h77fa898_7\",\n \"libgfortran-ng=13.2.0=h69a702a_7\",\n \"libgfortran5=13.2.0=hca663fb_7\",\n \"libgomp=13.2.0=h77fa898_7\",\n \"liblapack=3.9.0=22_linux64_openblas\",\n \"libnsl=2.0.1=hd590300_0\",\n \"libopenblas=0.3.27=pthreads_h413a1c8_0\",\n \"libprotobuf=4.25.3=h08a7969_0\",\n \"libsqlite=3.46.0=hde9e2c9_0\",\n \"libstdcxx-ng=13.2.0=hc0a3c3a_7\",\n \"libuuid=2.38.1=h0b41bf4_0\",\n \"libxcrypt=4.4.36=hd590300_1\",\n \"libzlib=1.3.1=h4ab18f5_1\",\n \"loguru=0.7.2=py310hff52083_1\",\n \"markdown-it-py=3.0.0=pyhd8ed1ab_0\",\n \"mdurl=0.1.2=pyhd8ed1ab_0\",\n \"mlflow-skinny=2.11.2=py310hff52083_0\",\n \"ncurses=6.5=h59595ed_0\",\n \"nltk=3.8.1=pyhd8ed1ab_0\",\n \"numpy=1.26.4=py310hb13e2d6_0\",\n \"openssl=3.3.1=h4ab18f5_0\",\n \"packaging=23.2=pyhd8ed1ab_0\",\n \"pandas=2.2.2=py310hf9f9076_1\",\n \"pip=24.0=pyhd8ed1ab_0\",\n \"pluggy=1.5.0=pyhd8ed1ab_0\",\n \"protobuf=4.25.3=py310ha8c1f0e_0\",\n \"pydantic=2.7.3=pyhd8ed1ab_0\",\n \"pydantic-core=2.18.4=py310he421c4c_0\",\n \"pydantic-settings=2.3.2=pyhd8ed1ab_0\",\n \"pygments=2.18.0=pyhd8ed1ab_0\",\n \"pysocks=1.7.1=pyha2e5f31_6\",\n \"pytest=8.2.2=pyhd8ed1ab_0\",\n \"python=3.10.14=hd12c33a_0_cpython\",\n \"python-dateutil=2.9.0=pyhd8ed1ab_0\",\n \"python-dotenv=1.0.1=pyhd8ed1ab_0\",\n \"python-tzdata=2024.1=pyhd8ed1ab_0\",\n \"python_abi=3.10=4_cp310\",\n \"pytz=2024.1=pyhd8ed1ab_0\",\n \"pyyaml=6.0.1=py310h2372a71_1\",\n \"readline=8.2=h8228510_1\",\n \"regex=2024.5.15=py310hc51659f_0\",\n \"requests=2.32.3=pyhd8ed1ab_0\",\n \"rich=13.7.1=pyhd8ed1ab_0\",\n \"scikit-learn=1.4.2=py310h981052a_1\",\n \"scipy=1.13.1=py310h93e2701_0\",\n \"setuptools=70.0.0=pyhd8ed1ab_0\",\n \"shellingham=1.5.4=pyhd8ed1ab_0\",\n \"six=1.16.0=pyh6c4a22f_0\",\n \"smmap=5.0.0=pyhd8ed1ab_0\",\n \"sqlparse=0.5.0=pyhd8ed1ab_0\",\n \"threadpoolctl=3.5.0=pyhc1e730c_0\",\n \"tk=8.6.13=noxft_h4845f30_101\",\n \"tomli=2.0.1=pyhd8ed1ab_0\",\n \"tqdm=4.66.4=pyhd8ed1ab_0\",\n \"typer=0.12.3=pyhd8ed1ab_0\",\n \"typer-slim=0.12.3=pyhd8ed1ab_0\",\n \"typer-slim-standard=0.12.3=hd8ed1ab_0\",\n \"typing-extensions=4.12.2=hd8ed1ab_0\",\n \"typing_extensions=4.12.2=pyha770c72_0\",\n \"tzdata=2024a=h0c530f3_0\",\n \"urllib3=2.2.1=pyhd8ed1ab_0\",\n \"wheel=0.43.0=pyhd8ed1ab_1\",\n \"xz=5.2.6=h166bdaf_0\",\n \"yaml=0.2.5=h7f98852_2\",\n \"zipp=3.19.2=pyhd8ed1ab_0\",\n {\n \"pip\": [\n \"azure-core==1.30.2\",\n \"azure-identity==1.16.1\",\n \"azureml-inference-server-http==1.3.0\",\n \"azure-monitor-opentelemetry==1.4.2\",\n \"azure-monitor-opentelemetry-exporter==1.0.0b25\",\n \"blinker==1.8.2\",\n \"cachetools==5.3.3\",\n \"cffi==1.16.0\",\n \"cryptography==42.0.8\",\n \"flask==2.3.2\",\n \"flask-cors==3.0.10\",\n \"google-api-core==2.19.0\",\n \"google-auth==2.30.0\",\n \"googleapis-common-protos==1.63.1\",\n \"gunicorn==22.0.0\",\n \"inference-schema==1.8\",\n \"itsdangerous==2.2.0\",\n \"jinja2==3.1.4\",\n \"markupsafe==2.1.5\",\n \"msal==1.28.1\",\n \"msal-extensions==1.1.0\",\n \"opencensus==0.11.4\",\n \"opencensus-context==0.1.3\",\n \"opencensus-ext-azure==1.1.13\",\n \"portalocker==2.8.2\",\n \"proto-plus==1.23.0\",\n \"psutil==5.9.8\",\n \"pyasn1==0.6.0\",\n \"pyasn1-modules==0.4.0\",\n \"pycparser==2.22\",\n \"pyjwt==2.8.0\",\n \"rsa==4.9\",\n \"werkzeug==3.0.3\",\n \"wrapt==1.16.0\"\n ]\n }\n ],\n \"name\": \"mlops-deploy-dc\",\n \"prefix\": \"/home/robin/miniforge3/envs/mlops-deploy-dc\"\n}", 2024-10-18T15:40:07.5418520Z "osType": "Linux", 2024-10-18T15:40:07.5418773Z "autoRebuild": "Disabled", 2024-10-18T15:40:07.5419034Z "provisioningState": "Succeeded", 2024-10-18T15:40:07.5419291Z "stage": "Development", 2024-10-18T15:40:07.5419544Z "intellectualProperty": null 2024-10-18T15:40:07.5419768Z }, 2024-10-18T15:40:07.5419998Z "systemData": { 2024-10-18T15:40:07.5420399Z "createdAt": "2024-10-18T15:39:26.9387438+00:00", 2024-10-18T15:40:07.5420854Z "createdBy": "", 2024-10-18T15:40:07.5421121Z "createdByType": "Application", 2024-10-18T15:40:07.5421544Z "lastModifiedAt": "2024-10-18T15:39:26.9387438+00:00", 2024-10-18T15:40:07.5421998Z "lastModifiedBy": "", 2024-10-18T15:40:07.5422281Z "lastModifiedByType": "Application" 2024-10-18T15:40:07.5422528Z } 2024-10-18T15:40:07.5422721Z } 2024-10-18T15:40:07.5422913Z ] 2024-10-18T15:40:07.5423101Z } 2024-10-18T15:40:07.5494901Z DEBUG: msal.telemetry: Generate or reuse correlation_id: 1f7b372c-ca07-4042-9124-5c515ff05085 2024-10-18T15:40:07.5495959Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces/?api-version=2024-07-01-preview' 2024-10-18T15:40:07.5496742Z Request method: 'GET' 2024-10-18T15:40:07.5496981Z Request headers: 2024-10-18T15:40:07.5497307Z 'Accept': 'application/json' 2024-10-18T15:40:07.5497728Z 'x-ms-client-request-id': '39efbce9-8d67-11ef-b286-75264f4eb5e7' 2024-10-18T15:40:07.5498549Z 'User-Agent': 'azureml-cli-v2/2.30.0 azure-ai-ml/1.20.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.8 (Linux-5.15.0-1073-azure-x86_64-with-glibc2.31) VSTS_260e2ed3-f14f-4b76-a777-9194773130c5_build_632_0' 2024-10-18T15:40:07.5502316Z 'Authorization': 'Bearer 2024-10-18T15:40:07.5502559Z Request body: 2024-10-18T15:40:07.5502797Z This request has no body 2024-10-18T15:40:07.5538052Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 2024-10-18T15:40:07.6224765Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces/?api-version=2024-07-01-preview HTTP/1.1" 404 116 2024-10-18T15:40:07.6225818Z DEBUG: azure.core.pipeline.policies._universal: Response status: '404' 2024-10-18T15:40:07.6226771Z Response headers: 2024-10-18T15:40:07.6227819Z 'Cache-Control': 'no-cache' 2024-10-18T15:40:07.6228193Z 'Pragma': 'no-cache' 2024-10-18T15:40:07.6228521Z 'Content-Length': '116' 2024-10-18T15:40:07.6228975Z 'Content-Type': 'application/json; charset=utf-8' 2024-10-18T15:40:07.6229321Z 'Expires': '-1' 2024-10-18T15:40:07.6229643Z 'x-ms-failure-cause': 'gateway' 2024-10-18T15:40:07.6230059Z 'x-ms-request-id': '5e92ee52-4081-4196-b239-f32fc3bb7e44' 2024-10-18T15:40:07.6230525Z 'x-ms-correlation-request-id': '5e92ee52-4081-4196-b239-f32fc3bb7e44' 2024-10-18T15:40:07.6231079Z 'x-ms-routing-request-id': 'WESTEUROPE:20241018T154007Z:5e92ee52-4081-4196-b239-f32fc3bb7e44' 2024-10-18T15:40:07.6231603Z 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' 2024-10-18T15:40:07.6231993Z 'X-Content-Type-Options': 'nosniff' 2024-10-18T15:40:07.6232342Z 'X-Cache': 'CONFIG_NOCACHE' 2024-10-18T15:40:07.6232866Z 'X-MSEdge-Ref': 'Ref A: 2112167B5FB1475EA971028364F8EBCB Ref B: AMS231020512027 Ref C: 2024-10-18T15:40:07Z' 2024-10-18T15:40:07.6233318Z 'Date': 'Fri, 18 Oct 2024 15:40:07 GMT' 2024-10-18T15:40:07.6233583Z Response content: 2024-10-18T15:40:07.6234135Z {"error":{"code":"ResourceGroupNotFound","message":"Resource group '' could not be found."}} 2024-10-18T15:40:07.6594727Z Traceback (most recent call last): 2024-10-18T15:40:07.6603566Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/online_deployment.py", line 118, in ml_online_deployment_create 2024-10-18T15:40:07.6604594Z deployment = ml_client.begin_create_or_update( 2024-10-18T15:40:07.6604991Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6605500Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_ml_client.py", line 1257, in begin_create_or_update 2024-10-18T15:40:07.6606051Z return _begin_create_or_update(entity, self._operation_container.all_operations, kwargs) 2024-10-18T15:40:07.6606440Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6606777Z File "/opt/az/lib/python3.11/functools.py", line 909, in wrapper 2024-10-18T15:40:07.6607172Z return dispatch(args[0].class)(args, kw) 2024-10-18T15:40:07.6607483Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6615977Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_mlclient.py", line 1363, in 2024-10-18T15:40:07.6616741Z return operations[AzureMLResourceType.ONLINE_DEPLOYMENT].begin_create_or_update(entity, kwargs) 2024-10-18T15:40:07.6617297Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6618755Z File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer 2024-10-18T15:40:07.6619511Z return func(args, kwargs) 2024-10-18T15:40:07.6619932Z ^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6620450Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 292, in wrapper 2024-10-18T15:40:07.6620951Z return f(*args, kwargs) 2024-10-18T15:40:07.6621388Z ^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6622268Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_deployment_operations.py", line 218, in begin_create_or_update 2024-10-18T15:40:07.6622755Z raise ex 2024-10-18T15:40:07.6623222Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_deployment_operations.py", line 213, in begin_create_or_update 2024-10-18T15:40:07.6623675Z raise ex 2024-10-18T15:40:07.6624131Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_deployment_operations.py", line 188, in begin_create_or_update 2024-10-18T15:40:07.6624988Z location = self._get_workspace_location() 2024-10-18T15:40:07.6625292Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6625740Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_online_deployment_operations.py", line 377, in _get_workspace_location 2024-10-18T15:40:07.6626312Z self._all_operations.all_operations[AzureMLResourceType.WORKSPACE].get(self._workspace_name).location 2024-10-18T15:40:07.6626700Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6627118Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 292, in wrapper 2024-10-18T15:40:07.6627563Z return f(*args, *kwargs) 2024-10-18T15:40:07.6627836Z ^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6628547Z File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer 2024-10-18T15:40:07.6628988Z return func(args, kwargs) 2024-10-18T15:40:07.6629264Z ^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6629681Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_workspace_operations.py", line 141, in get 2024-10-18T15:40:07.6630148Z return super().get(workspace_name=name, kwargs) 2024-10-18T15:40:07.6630467Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6630901Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_workspace_operations_base.py", line 87, in get 2024-10-18T15:40:07.6631380Z obj = self._operation.get(resource_group, workspace_name) 2024-10-18T15:40:07.6631713Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6632274Z File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer 2024-10-18T15:40:07.6632668Z return func(*args, *kwargs) 2024-10-18T15:40:07.6632950Z ^^^^^^^^^^^^^^^^^^^^^ 2024-10-18T15:40:07.6633409Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2024_07_01_preview/operations/_workspaces_operations.py", line 955, in get 2024-10-18T15:40:07.6633988Z map_error(status_code=response.status_code, response=response, error_map=error_map) 2024-10-18T15:40:07.6634836Z File "/opt/az/lib/python3.11/site-packages/azure/core/exceptions.py", line 112, in map_error 2024-10-18T15:40:07.6635193Z raise error 2024-10-18T15:40:07.6635914Z azure.core.exceptions.ResourceNotFoundError: (ResourceGroupNotFound) Resource group '' could not be found. 2024-10-18T15:40:07.6636325Z Code: ResourceGroupNotFound 2024-10-18T15:40:07.6637013Z Message: Resource group '' could not be found. 2024-10-18T15:40:07.6743633Z Code: ResourceGroupNotFound 2024-10-18T15:40:07.6744058Z Message: Resource group '' could not be found. 2024-10-18T15:40:07.6744207Z 2024-10-18T15:40:07.6744715Z ERROR: cli.azure.cli.core.azclierror: (ResourceGroupNotFound) Resource group '' could not be found. 2024-10-18T15:40:07.6745052Z Code: ResourceGroupNotFound 2024-10-18T15:40:07.6745474Z Message: Resource group '' could not be found. 2024-10-18T15:40:07.6746118Z ERROR: az_command_data_logger: (ResourceGroupNotFound) Resource group '' could not be found. 2024-10-18T15:40:07.6746440Z Code: ResourceGroupNotFound 2024-10-18T15:40:07.6746858Z Message: Resource group '' could not be found. 2024-10-18T15:40:07.6747982Z INFO: cli.main: Command ran in 11.295 seconds (init: 0.155, invoke: 11.140)

Expected behavior

The expected behaviour is that the RTS endpoint provisions using the Model in another subscription like how it used to work:

Environment Summary

{ "azure-cli": "2.64.0", "azure-cli-core": "2.64.0", "azure-cli-telemetry": "1.1.0", "extensions": { "azure-devops": "1.0.1" } }

az extension add -n ml --version 2.30.0

Additional context

In the --debug logs, the key piece of information is this:

2024-10-18T15:40:07.5495959Z DEBUG: azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/<Subscription A ID>/resourceGroups/prod-ml-endpoints/providers/Microsoft.MachineLearningServices/workspaces/***?api-version=2024-07-01-preview'

It's at this stage that the URI references the resource group from subscription B (prod-ml-endpoints) but with subscription A's ID instead. It's as if prior to this it was doing something with the model files in Subscription A and when it got to this part of the process it was erroneously referencing the wrong subscription ID.

Azure / azure-cli