search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.54k stars 2.77k forks source link

Can't deploy a model directly from AML registry to a workspace: InferenceContainerConfiguration.ServingContainer.Image: 'Image' must not be empty. #33598

Open alegend4u opened 9 months ago

alegend4u commented 9 months ago

Describe the bug I have an Azure Machine Learning Registry in which a model named model-A is registered. I have a KubernetesCompute attached to a workspace. The model isn't registered in the workspace. When I try to deploy the model to the workspace directly from the registry, it gives me following error (full traceback provided later): (DeploymentFailed) InferencingClient HttpRequest error, error detail: {"errors":{"InferenceContainerConfiguration.ServingContainer.Image":["'Image' must not be empty."]},"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","title":"One or more validation errors occurred.","status":400,"traceId":"XX-3646XXXXd143XXXX985aXXXX442aXXXX-c3d9XXXX1801XXXX-XX"}

To Reproduce Steps to reproduce the behavior:

  1. Registered a model using local files. 
    model = Model(
name="model-A",
version="1",
path="/home/rishabh/model-A/"
)
ml_client_registry.models.create_or_update(model).result()

Here's the output of print(ml_client_registry.models.get(name='model-A', version='1')):

creation_context:
  created_at: '2023-12-16T16:00:24.463485+00:00'
  created_by: Rishabh Prajapati
  created_by_type: User
  last_modified_at: '2023-12-16T16:00:24.134076+00:00'
  last_modified_by: Rishabh Prajapati
  last_modified_by_type: User
id: azureml://registries/abcdef-pqr-xyz/models/model-A/versions/1
name: model-A
path: https://xxxxxxxxxx.blob.core.windows.net/abcdef-pqr-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/model-A
properties: {}
tags: {}
type: custom_model
version: '1'
  1. Create an endpoint 
    
endpoint = KubernetesOnlineEndpoint(
name=endpoint_name,
description=endpoint_description,
compute=k8s_compute_name,
auth_mode="key",
)

Create the endpoint

ml_client.online_endpoints.begin_create_or_update(endpoint).result()


3. Prepare model from registry and environment from workspace.

model = ml_client_registry.models.get(name='model-A', version="1") # Model from registry environment = ml_client_workspace.environments.get(name=model_environment_name, version=model_environment_version)

4. Create the deployment

deployment = KubernetesOnlineDeployment( name=deployment_name, endpoint_name=endpoint_name, model=model.id, environment=environment.id, code_configuration=CodeConfiguration( code=str(model_script_path.parent), scoring_script=model_script_path.name ), instance_type=instance_type, instance_count=instance_count, ) ml_client.online_deployments.begin_create_or_update(deployment).result() # This fails


Last statement fails with below traceback and exception:

OperationFailed Traceback (most recent call last) File ~/miniconda3/envs/rishabh/lib/python3.11/site-packages/azure/core/polling/base_polling.py:757, in LROBasePolling.run(self) 756 try: --> 757 self._poll() 759 except BadStatus as err:

File ~/miniconda3/envs/rishabh/lib/python3.11/site-packages/azure/core/polling/base_polling.py:789, in LROBasePolling._poll(self) 788 if _failed(self.status()): --> 789 raise OperationFailed("Operation failed or canceled") 791 final_get_url = self._operation.get_final_get_url(self._pipeline_response)

OperationFailed: Operation failed or canceled

The above exception was the direct cause of the following exception:

HttpResponseError Traceback (most recent call last) Cell In[38], line 2 1 # Create the deployment ----> 2 ml_client.online_deployments.begin_create_or_update(deployment).result() 3 print(f"Deployment {deployment_name} created.")

File ~/miniconda3/envs/rishabh/lib/python3.11/site-packages/azure/core/polling/_poller.py:251, in LROPoller.result(self, timeout) 242 def result(self, timeout: Optional[float] = None) -> PollingReturnType_co: 243 """Return the result of the long running operation, or 244 the result available after the specified timeout. 245 (...) 249 :raises ~azure.core.exceptions.HttpResponseError: Server problem with the query. 250 """ --> 251 self.wait(timeout) 252 return self._polling_method.resource()

File ~/miniconda3/envs/rishabh/lib/python3.11/site-packages/azure/core/tracing/decorator.py:78, in distributed_trace..decorator..wrapper_use_tracer(*args, *kwargs) 76 span_impl_type = settings.tracing_implementation() 77 if span_impl_type is None: ---> 78 return func(args, **kwargs) 80 # Merge span is parameter is set, but only if no explicit parent are passed 81 if merge_span and not passed_in_parent:

File ~/miniconda3/envs/rishabh/lib/python3.11/site-packages/azure/core/polling/_poller.py:270, in LROPoller.wait(self, timeout) 266 self._thread.join(timeout=timeout) 267 try: 268 # Let's handle possible None in forgiveness here 269 # https://github.com/python/mypy/issues/8165 --> 270 raise self._exception # type: ignore 271 except TypeError: # Was None 272 pass

File ~/miniconda3/envs/rishabh/lib/python3.11/site-packages/azure/core/polling/_poller.py:185, in LROPoller._start(self) 181 """Start the long running operation. 182 On completion, runs any callbacks. 183 """ 184 try: --> 185 self._polling_method.run() 186 except AzureError as error: 187 if not error.continuation_token:

File ~/miniconda3/envs/rishabh/lib/python3.11/site-packages/azure/core/polling/base_polling.py:772, in LROBasePolling.run(self) 765 raise HttpResponseError( 766 response=self._pipeline_response.http_response, 767 message=str(err), 768 error=err, 769 ) from err 771 except OperationFailed as err: --> 772 raise HttpResponseError(response=self._pipeline_response.http_response, error=err) from err

HttpResponseError: (DeploymentFailed) InferencingClient HttpRequest error, error detail: {"errors":{"InferenceContainerConfiguration.ServingContainer.Image":["'Image' must not be empty."]},"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","title":"One or more validation errors occurred.","status":400,"traceId":"XX-3646XXXXd143XXXX985aXXXX442aXXXX-c3d9XXXX1801XXXX-XX"} Code: DeploymentFailed Message: InferencingClient HttpRequest error, error detail: {"errors":{"InferenceContainerConfiguration.ServingContainer.Image":["'Image' must not be empty."]},"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","title":"One or more validation errors occurred.","status":400,"traceId":"XX-3646XXXXd143XXXX985aXXXX442aXXXX-c3d9XXXX1801XXXX-XX"} Exception Details: (DeploymentFailed) InferencingClient HttpRequest error, error detail: {"errors":{"InferenceContainerConfiguration.ServingContainer.Image":["'Image' must not be empty."]},"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","title":"One or more validation errors occurred.","status":400,"traceId":"XX-3646XXXXd143XXXX985aXXXX442aXXXX-c3d9XXXX1801XXXX-XX"} Code: DeploymentFailed Message: InferencingClient HttpRequest error, error detail: {"errors":{"InferenceContainerConfiguration.ServingContainer.Image":["'Image' must not be empty."]},"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1","title":"One or more validation errors occurred.","status":400,"traceId":"XX-3646XXXXd143XXXX985aXXXX442aXXXX-c3d9XXXX1801XXXX-XX"}


 
**Expected behavior**
There shouldn't be any occur.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
- See "Deploy the model from registry to the online endpoint" section in this [official tutorial notebook](https://github.com/Azure/azureml-examples/blob/cfb51c4a5b357c79ee5bdf66993705b98a52a6fa/sdk/python/assets/assets-in-registry/share-models-components-environments.ipynb). It does the same thing I am trying to do.

- There's no direct way to register a model or environment from ML registry to workspace.
YalinLi0312 commented 9 months ago

Hi @alegend4u thanks for your contact, we'll investigate ASAP.

@azureml-github Can anyone in ml team help on it?

github-actions[bot] commented 9 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

github-actions[bot] commented 9 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

banibrata commented 9 months ago

Could you please share the debug logs for step number 4:, make sure any privacy data (e.g. accessToken is removed before sharing)

For SDK: If you are using the SDK directly, you can enable debug logs by following these steps:

Make sure that when the ml_client is instantiated, the logging_enable parameter is set to True. For example, ml_client = MLClient(..., logging_enable=True). Just before the call that is failing in the SDK, insert the following code: import logging logging.basicConfig(level=logging.DEBUG)

alegend4u commented 9 months ago

@banibrata, Thanks for responding.

Here are the debug logs:

Check: endpoint <endpoint_name> exists
DEBUG:azure.core.pipeline.policies._universal:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/onlineEndpoints/<endpoint_name>?api-version=2023-04-01-preview'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '<x-ms-client-request-id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'Bearer <auth_token>'
Request body:
This request has no body
INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/onlineEndpoints/<endpoint_name>?api-version=REDACTED'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '<x-ms-client-request-id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'REDACTED'
No body was attached to the request
DEBUG:urllib3.connectionpool:https://management.azure.com:443 "GET /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/onlineEndpoints/<endpoint_name>?api-version=2023-04-01-preview HTTP/1.1" 200 None
INFO:azure.core.pipeline.policies.http_logging_policy:Response status: 200
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'REDACTED'
    'Expires': '-1'
    'Vary': 'REDACTED'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11998'
    'Request-Context': 'REDACTED'
    'x-ms-response-type': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'x-aml-cluster': 'REDACTED'
    'x-request-time': 'REDACTED'
    'x-ms-correlation-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Date': 'Tue, 19 Dec 2023 17:42:33 GMT'
DEBUG:azure.core.pipeline.policies._universal:Response status: '200'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'gzip'
    'Expires': '-1'
    'Vary': 'Accept-Encoding,Accept-Encoding'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11998'
    'Request-Context': 'appId=cid-v1:<some_id>'
    'x-ms-response-type': 'standard'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-<location>-01'
    'x-request-time': '0.022'
    'x-ms-correlation-request-id': '<x-ms-request-id>'
    'x-ms-routing-request-id': 'CANADAEAST:20231219T174233Z:<x-ms-request-id>'
    'Date': 'Tue, 19 Dec 2023 17:42:33 GMT'
Response content:
{
  "id": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/onlineEndpoints/<endpoint_name>",
  "name": "<endpoint_name>",
  "type": "Microsoft.MachineLearningServices/workspaces/onlineEndpoints",
  "properties": {
    "description": "",
    "properties": {
      "azureml.onlineendpointid": "/subscriptions/<subscription_id>/resourcegroups/<resource_group>/providers/microsoft.machinelearningservices/workspaces/<workspace_name>/onlineendpoints/<endpoint_name>",
      "AzureAsyncOperationUri": "https://management.azure.com/subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/oe:<some_id>:<some_id>?api-version=2023-04-01-preview"
    },
    "scoringUri": "http://20.237.66.182/api/v1/endpoint/<endpoint_name>/score",
    "swaggerUri": "http://20.237.66.182/api/v1/endpoint/<endpoint_name>/swagger.json",
    "authMode": "Key",
    "provisioningState": "Succeeded",
    "compute": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/computes/<k8s_compute_name>",
    "publicNetworkAccess": "Enabled",
    "traffic": {},
    "mirrorTraffic": {}
  },
  "systemData": {
    "createdAt": "2023-12-19T17:41:27.1285398+00:00",
    "createdBy": "Some User",
    "lastModifiedAt": "2023-12-19T17:41:27.1285398+00:00"
  },
  "tags": {},
  "location": "<location>",
  "kind": "K8S",
  "identity": {
    "type": "SystemAssigned",
    "principalId": "<principal_id>",
    "tenantId": "<tenant_id>"
  }
}
DEBUG:azure.identity._internal.decorators:AzureCliCredential.get_token succeeded
DEBUG:azure.identity._internal.decorators:[Authenticated account] Client ID: <some_id>. Tenant ID: <tenant_id>. User Principal Name: <user.email@microsoft.com>. Object ID (user): <object_id>
INFO:azure.identity._credentials.default:DefaultAzureCredential acquired a token from AzureCliCredential
DEBUG:azure.core.pipeline.policies._universal:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<code_id>/versions?api-version=2023-04-01&hash=eceb86e2f8a30565a5abb6c66d072b8408425d05d25682a7f35edf37d2834290&hashVersion=202208'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'Bearer <auth_token>'
Request body:
This request has no body
INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<code_id>/versions?api-version=REDACTED&hash=REDACTED&hashVersion=REDACTED'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'REDACTED'
No body was attached to the request
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): management.azure.com:443
DEBUG:urllib3.connectionpool:https://management.azure.com:443 "GET /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<code_id>/versions?api-version=2023-04-01&hash=eceb86e2f8a30565a5abb6c66d072b8408425d05d25682a7f35edf37d2834290&hashVersion=202208 HTTP/1.1" 200 None
INFO:azure.core.pipeline.policies.http_logging_policy:Response status: 200
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'REDACTED'
    'Expires': '-1'
    'Vary': 'REDACTED'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11999'
    'Request-Context': 'REDACTED'
    'x-ms-response-type': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'x-aml-cluster': 'REDACTED'
    'x-request-time': 'REDACTED'
    'x-ms-correlation-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Date': 'Tue, 19 Dec 2023 17:42:34 GMT'
DEBUG:azure.core.pipeline.policies._universal:Response status: '200'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'gzip'
    'Expires': '-1'
    'Vary': 'Accept-Encoding,Accept-Encoding'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11999'
    'Request-Context': 'appId=cid-v1:<some_id>'
    'x-ms-response-type': 'standard'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-<location>-01'
    'x-request-time': '0.275'
    'x-ms-correlation-request-id': '<some_id>'
    'x-ms-routing-request-id': 'CANADAEAST:20231219T174234Z:<some_id>'
    'Date': 'Tue, 19 Dec 2023 17:42:34 GMT'
Response content:
{
  "value": [
    {
      "id": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<some_id>/versions/1",
      "name": "1",
      "type": "Microsoft.MachineLearningServices/workspaces/codes/versions",
      "properties": {
        "description": null,
        "tags": {},
        "properties": {
          "hash_sha256": "eceb86e2f8a30565a5abb6c66d072b8408425d05d25682a7f35edf37d2834290",
          "hash_version": "202208"
        },
        "isArchived": false,
        "isAnonymous": false,
        "codeUri": "https://<storage_account_id>.blob.core.windows.net:443/<some_id>-qi92udh3xsrszrqtphp3dmaivb/code",
        "provisioningState": "Succeeded"
      },
      "systemData": {
        "createdAt": "2023-12-17T05:03:52.2713804+00:00",
        "createdBy": "Some User",
        "createdByType": "User",
        "lastModifiedAt": "2023-12-17T05:03:52.2713804+00:00",
        "lastModifiedBy": "Some User",
        "lastModifiedByType": "User"
      }
    }
  ]
}
DEBUG:azure.core.pipeline.policies._universal:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<some_id>/versions/1?api-version=2023-04-01'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'Bearer <auth_token>'
Request body:
This request has no body
INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<some_id>/versions/1?api-version=REDACTED'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'REDACTED'
No body was attached to the request
DEBUG:urllib3.connectionpool:https://management.azure.com:443 "GET /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<some_id>/versions/1?api-version=2023-04-01 HTTP/1.1" 200 None
INFO:azure.core.pipeline.policies.http_logging_policy:Response status: 200
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'REDACTED'
    'Expires': '-1'
    'Vary': 'REDACTED'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11998'
    'Request-Context': 'REDACTED'
    'x-ms-response-type': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'x-aml-cluster': 'REDACTED'
    'x-request-time': 'REDACTED'
    'x-ms-correlation-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Date': 'Tue, 19 Dec 2023 17:42:34 GMT'
DEBUG:azure.core.pipeline.policies._universal:Response status: '200'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'gzip'
    'Expires': '-1'
    'Vary': 'Accept-Encoding,Accept-Encoding'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11998'
    'Request-Context': 'appId=cid-v1:<some_id>'
    'x-ms-response-type': 'standard'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-<location>-01'
    'x-request-time': '0.122'
    'x-ms-correlation-request-id': '<some_id>'
    'x-ms-routing-request-id': 'CANADAEAST:20231219T174235Z:<some_id>'
    'Date': 'Tue, 19 Dec 2023 17:42:34 GMT'
Response content:
{
  "id": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<some_id>/versions/1",
  "name": "1",
  "type": "Microsoft.MachineLearningServices/workspaces/codes/versions",
  "properties": {
    "description": null,
    "tags": {},
    "properties": {
      "hash_sha256": "eceb86e2f8a30565a5abb6c66d072b8408425d05d25682a7f35edf37d2834290",
      "hash_version": "202208"
    },
    "isArchived": false,
    "isAnonymous": false,
    "codeUri": "https://<storage_account_id>.blob.core.windows.net:443/<some_id>-qi92udh3xsrszrqtphp3dmaivb/code",
    "provisioningState": "Succeeded"
  },
  "systemData": {
    "createdAt": "2023-12-17T05:03:52.2713804+00:00",
    "createdBy": "Some User",
    "createdByType": "User",
    "lastModifiedAt": "2023-12-17T05:03:52.2713804+00:00",
    "lastModifiedBy": "Some User",
    "lastModifiedByType": "User"
  }
}
DEBUG:azure.core.pipeline.policies._universal:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>?api-version=2023-08-01-preview'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'Bearer <auth_token>'
Request body:
This request has no body
INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>?api-version=REDACTED'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'REDACTED'
No body was attached to the request
DEBUG:urllib3.connectionpool:https://management.azure.com:443 "GET /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>?api-version=2023-08-01-preview HTTP/1.1" 200 None
INFO:azure.core.pipeline.policies.http_logging_policy:Response status: 200
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'REDACTED'
    'Expires': '-1'
    'Vary': 'REDACTED'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11997'
    'Request-Context': 'REDACTED'
    'x-ms-response-type': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'x-aml-cluster': 'REDACTED'
    'x-request-time': 'REDACTED'
    'x-ms-correlation-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Date': 'Tue, 19 Dec 2023 17:42:35 GMT'
DEBUG:azure.core.pipeline.policies._universal:Response status: '200'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'gzip'
    'Expires': '-1'
    'Vary': 'Accept-Encoding,Accept-Encoding'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11997'
    'Request-Context': 'appId=cid-v1:<some_id>'
    'x-ms-response-type': 'standard'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-<location>-01'
    'x-request-time': '0.024'
    'x-ms-correlation-request-id': '<some_id>'
    'x-ms-routing-request-id': 'CANADAEAST:20231219T174235Z:<some_id>'
    'Date': 'Tue, 19 Dec 2023 17:42:35 GMT'
Response content:
{
  "id": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>",
  "name": "<workspace_name>",
  "type": "Microsoft.MachineLearningServices/workspaces",
  "location": "<location>",
  "tags": {},
  "etag": null,
  "properties": {
    "friendlyName": "<workspace_name>",
    "description": "",
    "storageAccount": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Storage/storageAccounts/<storage_account_id>",
    "keyVault": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.Keyvault/vaults/<storage_account_id>",
    "applicationInsights": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.insights/components/testwgmainsights70b34fe5",
    "hbiWorkspace": false,
    "tenantId": "<tenant_id>",
    "imageBuildCompute": null,
    "provisioningState": "Succeeded",
    "managedNetwork": {
      "changeableIsolationModes": [
        "AllowInternetOutbound",
        "AllowOnlyApprovedOutbound"
      ],
      "isolationMode": "Disabled"
    },
    "v1LegacyMode": false,
    "softDeleteEnabled": false,
    "containerRegistry": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.ContainerRegistry/registries/<registry_name>",
    "creationTime": "2023-12-12T17:47:29.0380336Z",
    "notebookInfo": {
      "resourceId": "d02e9537980d4c149345b9eb96e1e1d5",
      "fqdn": "ml-<workspace_name>-<location>-<some_id>.<location>.notebooks.azure.net",
      "isPrivateLinkEnabled": false,
      "notebookPreparationError": null
    },
    "storageHnsEnabled": false,
    "workspaceId": "<some_id>",
    "linkedModelInventoryArmId": null,
    "privateLinkCount": 0,
    "publicNetworkAccess": "Enabled",
    "ipAllowlist": [],
    "discoveryUrl": "https://<location>.api.azureml.ms/discovery",
    "mlFlowTrackingUri": "azureml://<location>.api.azureml.ms/mlflow/v1.0/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>",
    "serverlessComputeSettings": null,
    "EnableSoftwareBillOfMaterials": false,
    "sdkTelemetryAppInsightsKey": "<some_id>",
    "enableDataIsolation": false,
    "azureOpenAI": {
      "provisioningState": "NotStarted",
      "resourceId": null,
      "endpoint": null,
      "proxyResourceId": null,
      "failureReason": null,
      "deployedModels": []
    },
    "azureCognitiveSearch": {
      "provisioningState": "NotStarted",
      "resourceId": null,
      "endpoint": null,
      "failureReason": null
    },
    "azureAIContentSafety": {
      "provisioningState": "NotStarted",
      "resourceId": null,
      "endpoint": null,
      "failureReason": null
    },
    "endpointResources": {}
  },
  "identity": {
    "type": "SystemAssigned",
    "principalId": "<some_id>",
    "tenantId": "<tenant_id>"
  },
  "kind": "Default",
  "sku": {
    "name": "Basic",
    "tier": "Basic"
  },
  "systemData": {
    "createdAt": "2023-12-12T17:47:29.0380336Z",
    "createdBy": "<user.email@microsoft.com>",
    "createdByType": "User",
    "lastModifiedAt": "2023-12-12T17:47:29.0380336Z",
    "lastModifiedBy": "<user.email@microsoft.com>",
    "lastModifiedByType": "User"
  }
}
DEBUG:azure.core.pipeline.policies._universal:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/onlineEndpoints/<endpoint_name>/deployments/<model_deployment_name>?api-version=2023-04-01-preview'
Request method: 'PUT'
Request headers:
    'Content-Type': 'application/json'
    'Content-Length': '807'
    'Accept': 'application/json'
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'Bearer <auth_token>'
Request body:
{"tags": {}, "location": "<location>", "properties": {"codeConfiguration": {"codeId": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<some_id>/versions/1", "scoringScript": "run.py"}, "environmentId": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/environments/<environment_name>/versions/1", "environmentVariables": {}, "properties": {}, "appInsightsEnabled": false, "endpointComputeType": "Kubernetes", "instanceType": "wgminstancetype", "model": "azureml://registries/<global_registry_name>/models/<model_in_registry>/versions/1"}, "sku": {"name": "Default", "capacity": 1}}
INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/onlineEndpoints/<endpoint_name>/deployments/<model_deployment_name>?api-version=REDACTED'
Request method: 'PUT'
Request headers:
    'Content-Type': 'application/json'
    'Content-Length': '807'
    'Accept': 'application/json'
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'REDACTED'
A body is sent with the request
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): westus2-0.in.applicationinsights.azure.com:443
DEBUG:urllib3.connectionpool:https://westus2-0.in.applicationinsights.azure.com:443 "POST /v2.1/track HTTP/1.1" 200 49
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): westus2-0.in.applicationinsights.azure.com:443
DEBUG:urllib3.connectionpool:https://management.azure.com:443 "PUT /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/onlineEndpoints/<endpoint_name>/deployments/<model_deployment_name>?api-version=2023-04-01-preview HTTP/1.1" 201 2732
INFO:azure.core.pipeline.policies.http_logging_policy:Response status: 201
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Content-Length': '2732'
    'Content-Type': 'application/json; charset=utf-8'
    'Expires': '-1'
    'Location': 'REDACTED'
    'x-ms-ratelimit-remaining-subscription-resource-requests': '24'
    'Request-Context': 'REDACTED'
    'x-ms-response-type': 'REDACTED'
    'Azure-AsyncOperation': 'REDACTED'
    'x-ms-async-operation-timeout': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'x-aml-cluster': 'REDACTED'
    'x-request-time': 'REDACTED'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-correlation-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Date': 'Tue, 19 Dec 2023 17:42:37 GMT'
DEBUG:azure.core.pipeline.policies._universal:Response status: '201'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Content-Length': '2732'
    'Content-Type': 'application/json; charset=utf-8'
    'Expires': '-1'
    'Location': 'https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/onlineEndpoints/<endpoint_name>/deployments/<model_deployment_name>?api-version=2023-04-01-preview'
    'x-ms-ratelimit-remaining-subscription-resource-requests': '24'
    'Request-Context': 'appId=cid-v1:<some_id>'
    'x-ms-response-type': 'standard'
    'Azure-AsyncOperation': 'https://management.azure.com/subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>?api-version=2023-04-01-preview&t=638386045579519800&c=MIIHHjCCBgagAwIBAgITOgI8etKSzaPU5OVqYQAEAjx60jANBgkqhkiG9w0BAQsFADBEMRMwEQYKCZImiZPyLGQBGRYDR0JMMRMwEQYKCZImiZPyLGQBGRYDQU1FMRgwFgYDVQQDEw9BTUUgSU5GUkEgQ0EgMDEwHhcNMjMxMTAxMDUwMTE3WhcNMjQxMDI2MDUwMTE3WjBAMT4wPAYDVQQDEzVhc3luY29wZXJhdGlvbnNpZ25pbmdjZXJ0aWZpY2F0ZS5tYW5hZ2VtZW50LmF6dXJlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALfxA6qNFrWppPDpgIW0Za6jz8dJbtx-0kFRx9-xfDHr_HLBm4evdL9LREDW4KEhHrjGGXazAv7VzV0K50QeRNT5kZS6Sg1x7odrOpZfprkD9_jR119bgqtgHOo3p0UOUI5OFfo7p4R1u4PZrKrbYaAgCa7VdX1uQ7Li6YjiuSDU_klBFNnsK9jAasT4V8C9_dHfevkm8-S6hNulPvrvFl4dv8f9OSqolwSGeiLu2Txo8uh25LFhMsIdzGe68Atm8x51k1I8G3T_opuQ5l3bXtL74Marv79iFk9aF4Ll4wcYhZXY99YQo_L0HIccFMlhyPtbSnLD-A0KuJS9QSAdDe0CAwEAAaOCBAswggQHMCcGCSsGAQQBgjcVCgQaMBgwCgYIKwYBBQUHAwEwCgYIKwYBBQUHAwIwPQYJKwYBBAGCNxUHBDAwLgYmKwYBBAGCNxUIhpDjDYTVtHiE8Ys-hZvdFs6dEoFggvX2K4Py0SACAWQCAQowggHaBggrBgEFBQcBAQSCAcwwggHIMGYGCCsGAQUFBzAChlpodHRwOi8vY3JsLm1pY3Jvc29mdC5jb20vcGtpaW5mcmEvQ2VydHMvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmwxLmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MFYGCCsGAQUFBzAChkpodHRwOi8vY3JsMi5hbWUuZ2JsL2FpYS9CWTJQS0lJTlRDQTAxLkFNRS5HQkxfQU1FJTIwSU5GUkElMjBDQSUyMDAxKDQpLmNydDBWBggrBgEFBQcwAoZKaHR0cDovL2NybDMuYW1lLmdibC9haWEvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmw0LmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MB0GA1UdDgQWBBQR6ugmtGgP0QrK4G_av0HgY6W8TzAOBgNVHQ8BAf8EBAMCBaAwggE1BgNVHR8EggEsMIIBKDCCASSgggEgoIIBHIZCaHR0cDovL2NybC5taWNyb3NvZnQuY29tL3BraWluZnJhL0NSTC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMS5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMi5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMy5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsNC5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JsMBcGA1UdIAQQMA4wDAYKKwYBBAGCN3sBATAfBgNVHSMEGDAWgBTl2Ztn_PjsurvwwKidileIud8-YzAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDQYJKoZIhvcNAQELBQADggEBADqxubH0GMcJ8aWU7jZie4uu3Js-ArE1T9MSRSdVz7UtaYNpuDte9Ki4puBDPhJgO7-2ptyQy1bc1Bvj392Miq7Xdnl95e88M5lggNFnb9Tb1FNGUtH1b9KNiby9-xBeOcXkOCKagXNFv3D16-8GdLjTdb1rLwhDz77SVI8A2ikg5NcDVjKeRhW1UQ7_pC8ZR1L9hmdxJ_dkzMZ45kVvxlFRtkOBdxD1xIVdqyoKYFNpX63aoItuCiu5TXK3rS3nCXlAOwIFtpGG9MPwVu-QntqmkZe8lRWpQPGoPIodyUg5vNJj8V_u3NwbydLGD92yiDA3rE0JAfT9_6jr0pLIJRc&s=qwdrWVfsInoceQeY99e0O-_FcpS0CJ7Wceyc9QfUVhRXcZGgVO0CSiZyU32EQACF-VO3PvJ3tzBIl2UJ3AXknEitPt2yRF_G8OIUscPiP4k9W0RP6jujTt4jwyRB58a-_sY5vCaUGRjCzlXlVgBj_gjVZ9dePkay0hqO_aKAcKZbP39IXWXBFiCh06sGh2YuSbfGsROjU_Efc0ZJTs_rKaaHsOXm4C2sUdAAVCCxGKRXiEXT65mZcZl1v55tYsTdVOAqKqc0q3NlPmTax5jd_nxBUZjDrhdjLfIQXpuk5hIPstlCtvsDeLRQnXLWQ3XLzpORf7ydSgL_k0cL1jmuwA&h=iRm7sfii1BjJx8qUedxC56ZE0MbpiAGgt4JxMfX6_Vo'
    'x-ms-async-operation-timeout': 'PT30M'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-<location>-01'
    'x-request-time': '1.917'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-correlation-request-id': '<some_id>'
    'x-ms-routing-request-id': 'CANADAEAST:20231219T174237Z:<some_id>'
    'Date': 'Tue, 19 Dec 2023 17:42:37 GMT'
Response content:
{
  "id": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/onlineEndpoints/<endpoint_name>/deployments/<model_deployment_name>",
  "name": "<model_deployment_name>",
  "type": "Microsoft.MachineLearningServices/workspaces/onlineEndpoints/deployments",
  "properties": {
    "description": null,
    "properties": {
      "AzureAsyncOperationUri": "https://management.azure.com/subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>?api-version=2023-04-01-preview"
    },
    "codeConfiguration": {
      "codeId": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/codes/<some_id>/versions/1",
      "scoringScript": "run.py"
    },
    "environmentId": "/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/environments/<environment_name>/versions/1",
    "environmentVariables": {},
    "scaleSettings": {
      "scaleType": "TargetUtilization",
      "pollingInterval": "PT1S",
      "targetUtilizationPercentage": 70,
      "minInstances": 1,
      "maxInstances": 1
    },
    "requestSettings": {
      "maxQueueWait": "PT0.5S",
      "requestTimeout": "PT0S",
      "maxConcurrentRequestsPerInstance": 1
    },
    "modelMountPath": null,
    "appInsightsEnabled": false,
    "livenessProbe": {
      "failureThreshold": 30,
      "successThreshold": 1,
      "timeout": "PT2S",
      "period": "PT10S",
      "initialDelay": "PT10S"
    },
    "readinessProbe": {
      "failureThreshold": 30,
      "successThreshold": 1,
      "timeout": "PT2S",
      "period": "PT10S",
      "initialDelay": "PT10S"
    },
    "provisioningState": "Updating",
    "instanceType": "wgminstancetype",
    "model": "azureml://registries/<global_registry_name>/models/<model_in_registry>/versions/1",
    "egressPublicNetworkAccess": "Enabled",
    "dataCollector": null,
    "endpointComputeType": "Kubernetes",
    "containerResourceRequirements": {
      "containerResourceRequests": {
        "cpu": null,
        "memory": null,
        "gpu": null
      },
      "containerResourceLimits": {
        "cpu": null,
        "memory": null,
        "gpu": null
      }
    }
  },
  "systemData": {
    "createdAt": "2023-12-19T17:42:37.4673454+00:00",
    "createdBy": "Some User",
    "lastModifiedAt": "2023-12-19T17:42:37.4673454+00:00"
  },
  "tags": {},
  "location": "<location>",
  "kind": "K8S"
}
DEBUG:azure.core.pipeline.policies._universal:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>?api-version=2023-04-01-preview&t=638386045579519800&c=MIIHHjCCBgagAwIBAgITOgI8etKSzaPU5OVqYQAEAjx60jANBgkqhkiG9w0BAQsFADBEMRMwEQYKCZImiZPyLGQBGRYDR0JMMRMwEQYKCZImiZPyLGQBGRYDQU1FMRgwFgYDVQQDEw9BTUUgSU5GUkEgQ0EgMDEwHhcNMjMxMTAxMDUwMTE3WhcNMjQxMDI2MDUwMTE3WjBAMT4wPAYDVQQDEzVhc3luY29wZXJhdGlvbnNpZ25pbmdjZXJ0aWZpY2F0ZS5tYW5hZ2VtZW50LmF6dXJlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALfxA6qNFrWppPDpgIW0Za6jz8dJbtx-0kFRx9-xfDHr_HLBm4evdL9LREDW4KEhHrjGGXazAv7VzV0K50QeRNT5kZS6Sg1x7odrOpZfprkD9_jR119bgqtgHOo3p0UOUI5OFfo7p4R1u4PZrKrbYaAgCa7VdX1uQ7Li6YjiuSDU_klBFNnsK9jAasT4V8C9_dHfevkm8-S6hNulPvrvFl4dv8f9OSqolwSGeiLu2Txo8uh25LFhMsIdzGe68Atm8x51k1I8G3T_opuQ5l3bXtL74Marv79iFk9aF4Ll4wcYhZXY99YQo_L0HIccFMlhyPtbSnLD-A0KuJS9QSAdDe0CAwEAAaOCBAswggQHMCcGCSsGAQQBgjcVCgQaMBgwCgYIKwYBBQUHAwEwCgYIKwYBBQUHAwIwPQYJKwYBBAGCNxUHBDAwLgYmKwYBBAGCNxUIhpDjDYTVtHiE8Ys-hZvdFs6dEoFggvX2K4Py0SACAWQCAQowggHaBggrBgEFBQcBAQSCAcwwggHIMGYGCCsGAQUFBzAChlpodHRwOi8vY3JsLm1pY3Jvc29mdC5jb20vcGtpaW5mcmEvQ2VydHMvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmwxLmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MFYGCCsGAQUFBzAChkpodHRwOi8vY3JsMi5hbWUuZ2JsL2FpYS9CWTJQS0lJTlRDQTAxLkFNRS5HQkxfQU1FJTIwSU5GUkElMjBDQSUyMDAxKDQpLmNydDBWBggrBgEFBQcwAoZKaHR0cDovL2NybDMuYW1lLmdibC9haWEvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmw0LmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MB0GA1UdDgQWBBQR6ugmtGgP0QrK4G_av0HgY6W8TzAOBgNVHQ8BAf8EBAMCBaAwggE1BgNVHR8EggEsMIIBKDCCASSgggEgoIIBHIZCaHR0cDovL2NybC5taWNyb3NvZnQuY29tL3BraWluZnJhL0NSTC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMS5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMi5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMy5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsNC5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JsMBcGA1UdIAQQMA4wDAYKKwYBBAGCN3sBATAfBgNVHSMEGDAWgBTl2Ztn_PjsurvwwKidileIud8-YzAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDQYJKoZIhvcNAQELBQADggEBADqxubH0GMcJ8aWU7jZie4uu3Js-ArE1T9MSRSdVz7UtaYNpuDte9Ki4puBDPhJgO7-2ptyQy1bc1Bvj392Miq7Xdnl95e88M5lggNFnb9Tb1FNGUtH1b9KNiby9-xBeOcXkOCKagXNFv3D16-8GdLjTdb1rLwhDz77SVI8A2ikg5NcDVjKeRhW1UQ7_pC8ZR1L9hmdxJ_dkzMZ45kVvxlFRtkOBdxD1xIVdqyoKYFNpX63aoItuCiu5TXK3rS3nCXlAOwIFtpGG9MPwVu-QntqmkZe8lRWpQPGoPIodyUg5vNJj8V_u3NwbydLGD92yiDA3rE0JAfT9_6jr0pLIJRc&s=qwdrWVfsInoceQeY99e0O-_FcpS0CJ7Wceyc9QfUVhRXcZGgVO0CSiZyU32EQACF-VO3PvJ3tzBIl2UJ3AXknEitPt2yRF_G8OIUscPiP4k9W0RP6jujTt4jwyRB58a-_sY5vCaUGRjCzlXlVgBj_gjVZ9dePkay0hqO_aKAcKZbP39IXWXBFiCh06sGh2YuSbfGsROjU_Efc0ZJTs_rKaaHsOXm4C2sUdAAVCCxGKRXiEXT65mZcZl1v55tYsTdVOAqKqc0q3NlPmTax5jd_nxBUZjDrhdjLfIQXpuk5hIPstlCtvsDeLRQnXLWQ3XLzpORf7ydSgL_k0cL1jmuwA&h=iRm7sfii1BjJx8qUedxC56ZE0MbpiAGgt4JxMfX6_Vo'
Request method: 'GET'
Request headers:
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'Bearer <auth_token>'
Request body:
This request has no body
INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>?api-version=REDACTED&t=REDACTED&c=REDACTED&s=REDACTED&h=REDACTED'
Request method: 'GET'
Request headers:
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'REDACTED'
No body was attached to the request
DEBUG:urllib3.connectionpool:https://westus2-0.in.applicationinsights.azure.com:443 "POST /v2.1/track HTTP/1.1" 200 49
DEBUG:urllib3.connectionpool:https://management.azure.com:443 "GET /subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>?api-version=2023-04-01-preview&t=638386045579519800&c=MIIHHjCCBgagAwIBAgITOgI8etKSzaPU5OVqYQAEAjx60jANBgkqhkiG9w0BAQsFADBEMRMwEQYKCZImiZPyLGQBGRYDR0JMMRMwEQYKCZImiZPyLGQBGRYDQU1FMRgwFgYDVQQDEw9BTUUgSU5GUkEgQ0EgMDEwHhcNMjMxMTAxMDUwMTE3WhcNMjQxMDI2MDUwMTE3WjBAMT4wPAYDVQQDEzVhc3luY29wZXJhdGlvbnNpZ25pbmdjZXJ0aWZpY2F0ZS5tYW5hZ2VtZW50LmF6dXJlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALfxA6qNFrWppPDpgIW0Za6jz8dJbtx-0kFRx9-xfDHr_HLBm4evdL9LREDW4KEhHrjGGXazAv7VzV0K50QeRNT5kZS6Sg1x7odrOpZfprkD9_jR119bgqtgHOo3p0UOUI5OFfo7p4R1u4PZrKrbYaAgCa7VdX1uQ7Li6YjiuSDU_klBFNnsK9jAasT4V8C9_dHfevkm8-S6hNulPvrvFl4dv8f9OSqolwSGeiLu2Txo8uh25LFhMsIdzGe68Atm8x51k1I8G3T_opuQ5l3bXtL74Marv79iFk9aF4Ll4wcYhZXY99YQo_L0HIccFMlhyPtbSnLD-A0KuJS9QSAdDe0CAwEAAaOCBAswggQHMCcGCSsGAQQBgjcVCgQaMBgwCgYIKwYBBQUHAwEwCgYIKwYBBQUHAwIwPQYJKwYBBAGCNxUHBDAwLgYmKwYBBAGCNxUIhpDjDYTVtHiE8Ys-hZvdFs6dEoFggvX2K4Py0SACAWQCAQowggHaBggrBgEFBQcBAQSCAcwwggHIMGYGCCsGAQUFBzAChlpodHRwOi8vY3JsLm1pY3Jvc29mdC5jb20vcGtpaW5mcmEvQ2VydHMvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmwxLmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MFYGCCsGAQUFBzAChkpodHRwOi8vY3JsMi5hbWUuZ2JsL2FpYS9CWTJQS0lJTlRDQTAxLkFNRS5HQkxfQU1FJTIwSU5GUkElMjBDQSUyMDAxKDQpLmNydDBWBggrBgEFBQcwAoZKaHR0cDovL2NybDMuYW1lLmdibC9haWEvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmw0LmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MB0GA1UdDgQWBBQR6ugmtGgP0QrK4G_av0HgY6W8TzAOBgNVHQ8BAf8EBAMCBaAwggE1BgNVHR8EggEsMIIBKDCCASSgggEgoIIBHIZCaHR0cDovL2NybC5taWNyb3NvZnQuY29tL3BraWluZnJhL0NSTC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMS5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMi5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMy5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsNC5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JsMBcGA1UdIAQQMA4wDAYKKwYBBAGCN3sBATAfBgNVHSMEGDAWgBTl2Ztn_PjsurvwwKidileIud8-YzAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDQYJKoZIhvcNAQELBQADggEBADqxubH0GMcJ8aWU7jZie4uu3Js-ArE1T9MSRSdVz7UtaYNpuDte9Ki4puBDPhJgO7-2ptyQy1bc1Bvj392Miq7Xdnl95e88M5lggNFnb9Tb1FNGUtH1b9KNiby9-xBeOcXkOCKagXNFv3D16-8GdLjTdb1rLwhDz77SVI8A2ikg5NcDVjKeRhW1UQ7_pC8ZR1L9hmdxJ_dkzMZ45kVvxlFRtkOBdxD1xIVdqyoKYFNpX63aoItuCiu5TXK3rS3nCXlAOwIFtpGG9MPwVu-QntqmkZe8lRWpQPGoPIodyUg5vNJj8V_u3NwbydLGD92yiDA3rE0JAfT9_6jr0pLIJRc&s=qwdrWVfsInoceQeY99e0O-_FcpS0CJ7Wceyc9QfUVhRXcZGgVO0CSiZyU32EQACF-VO3PvJ3tzBIl2UJ3AXknEitPt2yRF_G8OIUscPiP4k9W0RP6jujTt4jwyRB58a-_sY5vCaUGRjCzlXlVgBj_gjVZ9dePkay0hqO_aKAcKZbP39IXWXBFiCh06sGh2YuSbfGsROjU_Efc0ZJTs_rKaaHsOXm4C2sUdAAVCCxGKRXiEXT65mZcZl1v55tYsTdVOAqKqc0q3NlPmTax5jd_nxBUZjDrhdjLfIQXpuk5hIPstlCtvsDeLRQnXLWQ3XLzpORf7ydSgL_k0cL1jmuwA&h=iRm7sfii1BjJx8qUedxC56ZE0MbpiAGgt4JxMfX6_Vo HTTP/1.1" 200 None
INFO:azure.core.pipeline.policies.http_logging_policy:Response status: 200
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'REDACTED'
    'Expires': '-1'
    'Vary': 'REDACTED'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11997'
    'Request-Context': 'REDACTED'
    'x-ms-response-type': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'x-aml-cluster': 'REDACTED'
    'x-request-time': 'REDACTED'
    'x-ms-correlation-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Date': 'Tue, 19 Dec 2023 17:42:38 GMT'
DEBUG:azure.core.pipeline.policies._universal:Response status: '200'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'gzip'
    'Expires': '-1'
    'Vary': 'Accept-Encoding,Accept-Encoding'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11997'
    'Request-Context': 'appId=cid-v1:<some_id>'
    'x-ms-response-type': 'standard'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-<location>-01'
    'x-request-time': '0.171'
    'x-ms-correlation-request-id': '<some_id>'
    'x-ms-routing-request-id': 'CANADAEAST:20231219T174238Z:<some_id>'
    'Date': 'Tue, 19 Dec 2023 17:42:38 GMT'
Response content:
{
  "id": "/subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>",
  "name": "od:<some_id>:<some_id>",
  "status": "Updating",
  "startTime": "2023-12-19T17:42:37.2440026+00:00",
  "percentComplete": 0.0
}
.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): westus2-0.in.applicationinsights.azure.com:443
DEBUG:urllib3.connectionpool:https://westus2-0.in.applicationinsights.azure.com:443 "POST /v2.1/track HTTP/1.1" 200 49
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): westus2-0.in.applicationinsights.azure.com:443
DEBUG:urllib3.connectionpool:https://westus2-0.in.applicationinsights.azure.com:443 "POST /v2.1/track HTTP/1.1" 200 49
DEBUG:azure.core.pipeline.policies._universal:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>?api-version=2023-04-01-preview&t=638386045579519800&c=MIIHHjCCBgagAwIBAgITOgI8etKSzaPU5OVqYQAEAjx60jANBgkqhkiG9w0BAQsFADBEMRMwEQYKCZImiZPyLGQBGRYDR0JMMRMwEQYKCZImiZPyLGQBGRYDQU1FMRgwFgYDVQQDEw9BTUUgSU5GUkEgQ0EgMDEwHhcNMjMxMTAxMDUwMTE3WhcNMjQxMDI2MDUwMTE3WjBAMT4wPAYDVQQDEzVhc3luY29wZXJhdGlvbnNpZ25pbmdjZXJ0aWZpY2F0ZS5tYW5hZ2VtZW50LmF6dXJlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALfxA6qNFrWppPDpgIW0Za6jz8dJbtx-0kFRx9-xfDHr_HLBm4evdL9LREDW4KEhHrjGGXazAv7VzV0K50QeRNT5kZS6Sg1x7odrOpZfprkD9_jR119bgqtgHOo3p0UOUI5OFfo7p4R1u4PZrKrbYaAgCa7VdX1uQ7Li6YjiuSDU_klBFNnsK9jAasT4V8C9_dHfevkm8-S6hNulPvrvFl4dv8f9OSqolwSGeiLu2Txo8uh25LFhMsIdzGe68Atm8x51k1I8G3T_opuQ5l3bXtL74Marv79iFk9aF4Ll4wcYhZXY99YQo_L0HIccFMlhyPtbSnLD-A0KuJS9QSAdDe0CAwEAAaOCBAswggQHMCcGCSsGAQQBgjcVCgQaMBgwCgYIKwYBBQUHAwEwCgYIKwYBBQUHAwIwPQYJKwYBBAGCNxUHBDAwLgYmKwYBBAGCNxUIhpDjDYTVtHiE8Ys-hZvdFs6dEoFggvX2K4Py0SACAWQCAQowggHaBggrBgEFBQcBAQSCAcwwggHIMGYGCCsGAQUFBzAChlpodHRwOi8vY3JsLm1pY3Jvc29mdC5jb20vcGtpaW5mcmEvQ2VydHMvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmwxLmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MFYGCCsGAQUFBzAChkpodHRwOi8vY3JsMi5hbWUuZ2JsL2FpYS9CWTJQS0lJTlRDQTAxLkFNRS5HQkxfQU1FJTIwSU5GUkElMjBDQSUyMDAxKDQpLmNydDBWBggrBgEFBQcwAoZKaHR0cDovL2NybDMuYW1lLmdibC9haWEvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmw0LmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MB0GA1UdDgQWBBQR6ugmtGgP0QrK4G_av0HgY6W8TzAOBgNVHQ8BAf8EBAMCBaAwggE1BgNVHR8EggEsMIIBKDCCASSgggEgoIIBHIZCaHR0cDovL2NybC5taWNyb3NvZnQuY29tL3BraWluZnJhL0NSTC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMS5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMi5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMy5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsNC5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JsMBcGA1UdIAQQMA4wDAYKKwYBBAGCN3sBATAfBgNVHSMEGDAWgBTl2Ztn_PjsurvwwKidileIud8-YzAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDQYJKoZIhvcNAQELBQADggEBADqxubH0GMcJ8aWU7jZie4uu3Js-ArE1T9MSRSdVz7UtaYNpuDte9Ki4puBDPhJgO7-2ptyQy1bc1Bvj392Miq7Xdnl95e88M5lggNFnb9Tb1FNGUtH1b9KNiby9-xBeOcXkOCKagXNFv3D16-8GdLjTdb1rLwhDz77SVI8A2ikg5NcDVjKeRhW1UQ7_pC8ZR1L9hmdxJ_dkzMZ45kVvxlFRtkOBdxD1xIVdqyoKYFNpX63aoItuCiu5TXK3rS3nCXlAOwIFtpGG9MPwVu-QntqmkZe8lRWpQPGoPIodyUg5vNJj8V_u3NwbydLGD92yiDA3rE0JAfT9_6jr0pLIJRc&s=qwdrWVfsInoceQeY99e0O-_FcpS0CJ7Wceyc9QfUVhRXcZGgVO0CSiZyU32EQACF-VO3PvJ3tzBIl2UJ3AXknEitPt2yRF_G8OIUscPiP4k9W0RP6jujTt4jwyRB58a-_sY5vCaUGRjCzlXlVgBj_gjVZ9dePkay0hqO_aKAcKZbP39IXWXBFiCh06sGh2YuSbfGsROjU_Efc0ZJTs_rKaaHsOXm4C2sUdAAVCCxGKRXiEXT65mZcZl1v55tYsTdVOAqKqc0q3NlPmTax5jd_nxBUZjDrhdjLfIQXpuk5hIPstlCtvsDeLRQnXLWQ3XLzpORf7ydSgL_k0cL1jmuwA&h=iRm7sfii1BjJx8qUedxC56ZE0MbpiAGgt4JxMfX6_Vo'
Request method: 'GET'
Request headers:
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'Bearer <auth_token>'
Request body:
This request has no body
INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://management.azure.com/subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>?api-version=REDACTED&t=REDACTED&c=REDACTED&s=REDACTED&h=REDACTED'
Request method: 'GET'
Request headers:
    'x-ms-client-request-id': '<some_id>'
    'User-Agent': 'azure-ai-ml/1.12.1 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.11.2 (Linux-5.15.0-88-generic-x86_64-with-glibc2.35)'
    'Authorization': 'REDACTED'
No body was attached to the request
DEBUG:urllib3.connectionpool:https://management.azure.com:443 "GET /subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>?api-version=2023-04-01-preview&t=638386045579519800&c=MIIHHjCCBgagAwIBAgITOgI8etKSzaPU5OVqYQAEAjx60jANBgkqhkiG9w0BAQsFADBEMRMwEQYKCZImiZPyLGQBGRYDR0JMMRMwEQYKCZImiZPyLGQBGRYDQU1FMRgwFgYDVQQDEw9BTUUgSU5GUkEgQ0EgMDEwHhcNMjMxMTAxMDUwMTE3WhcNMjQxMDI2MDUwMTE3WjBAMT4wPAYDVQQDEzVhc3luY29wZXJhdGlvbnNpZ25pbmdjZXJ0aWZpY2F0ZS5tYW5hZ2VtZW50LmF6dXJlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALfxA6qNFrWppPDpgIW0Za6jz8dJbtx-0kFRx9-xfDHr_HLBm4evdL9LREDW4KEhHrjGGXazAv7VzV0K50QeRNT5kZS6Sg1x7odrOpZfprkD9_jR119bgqtgHOo3p0UOUI5OFfo7p4R1u4PZrKrbYaAgCa7VdX1uQ7Li6YjiuSDU_klBFNnsK9jAasT4V8C9_dHfevkm8-S6hNulPvrvFl4dv8f9OSqolwSGeiLu2Txo8uh25LFhMsIdzGe68Atm8x51k1I8G3T_opuQ5l3bXtL74Marv79iFk9aF4Ll4wcYhZXY99YQo_L0HIccFMlhyPtbSnLD-A0KuJS9QSAdDe0CAwEAAaOCBAswggQHMCcGCSsGAQQBgjcVCgQaMBgwCgYIKwYBBQUHAwEwCgYIKwYBBQUHAwIwPQYJKwYBBAGCNxUHBDAwLgYmKwYBBAGCNxUIhpDjDYTVtHiE8Ys-hZvdFs6dEoFggvX2K4Py0SACAWQCAQowggHaBggrBgEFBQcBAQSCAcwwggHIMGYGCCsGAQUFBzAChlpodHRwOi8vY3JsLm1pY3Jvc29mdC5jb20vcGtpaW5mcmEvQ2VydHMvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmwxLmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MFYGCCsGAQUFBzAChkpodHRwOi8vY3JsMi5hbWUuZ2JsL2FpYS9CWTJQS0lJTlRDQTAxLkFNRS5HQkxfQU1FJTIwSU5GUkElMjBDQSUyMDAxKDQpLmNydDBWBggrBgEFBQcwAoZKaHR0cDovL2NybDMuYW1lLmdibC9haWEvQlkyUEtJSU5UQ0EwMS5BTUUuR0JMX0FNRSUyMElORlJBJTIwQ0ElMjAwMSg0KS5jcnQwVgYIKwYBBQUHMAKGSmh0dHA6Ly9jcmw0LmFtZS5nYmwvYWlhL0JZMlBLSUlOVENBMDEuQU1FLkdCTF9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3J0MB0GA1UdDgQWBBQR6ugmtGgP0QrK4G_av0HgY6W8TzAOBgNVHQ8BAf8EBAMCBaAwggE1BgNVHR8EggEsMIIBKDCCASSgggEgoIIBHIZCaHR0cDovL2NybC5taWNyb3NvZnQuY29tL3BraWluZnJhL0NSTC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMS5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMi5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsMy5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JshjRodHRwOi8vY3JsNC5hbWUuZ2JsL2NybC9BTUUlMjBJTkZSQSUyMENBJTIwMDEoNCkuY3JsMBcGA1UdIAQQMA4wDAYKKwYBBAGCN3sBATAfBgNVHSMEGDAWgBTl2Ztn_PjsurvwwKidileIud8-YzAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDQYJKoZIhvcNAQELBQADggEBADqxubH0GMcJ8aWU7jZie4uu3Js-ArE1T9MSRSdVz7UtaYNpuDte9Ki4puBDPhJgO7-2ptyQy1bc1Bvj392Miq7Xdnl95e88M5lggNFnb9Tb1FNGUtH1b9KNiby9-xBeOcXkOCKagXNFv3D16-8GdLjTdb1rLwhDz77SVI8A2ikg5NcDVjKeRhW1UQ7_pC8ZR1L9hmdxJ_dkzMZ45kVvxlFRtkOBdxD1xIVdqyoKYFNpX63aoItuCiu5TXK3rS3nCXlAOwIFtpGG9MPwVu-QntqmkZe8lRWpQPGoPIodyUg5vNJj8V_u3NwbydLGD92yiDA3rE0JAfT9_6jr0pLIJRc&s=qwdrWVfsInoceQeY99e0O-_FcpS0CJ7Wceyc9QfUVhRXcZGgVO0CSiZyU32EQACF-VO3PvJ3tzBIl2UJ3AXknEitPt2yRF_G8OIUscPiP4k9W0RP6jujTt4jwyRB58a-_sY5vCaUGRjCzlXlVgBj_gjVZ9dePkay0hqO_aKAcKZbP39IXWXBFiCh06sGh2YuSbfGsROjU_Efc0ZJTs_rKaaHsOXm4C2sUdAAVCCxGKRXiEXT65mZcZl1v55tYsTdVOAqKqc0q3NlPmTax5jd_nxBUZjDrhdjLfIQXpuk5hIPstlCtvsDeLRQnXLWQ3XLzpORf7ydSgL_k0cL1jmuwA&h=iRm7sfii1BjJx8qUedxC56ZE0MbpiAGgt4JxMfX6_Vo HTTP/1.1" 200 None
INFO:azure.core.pipeline.policies.http_logging_policy:Response status: 200
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'REDACTED'
    'Expires': '-1'
    'Vary': 'REDACTED'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11996'
    'Request-Context': 'REDACTED'
    'x-ms-response-type': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'x-aml-cluster': 'REDACTED'
    'x-request-time': 'REDACTED'
    'x-ms-correlation-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Date': 'Tue, 19 Dec 2023 17:42:43 GMT'
DEBUG:azure.core.pipeline.policies._universal:Response status: '200'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'gzip'
    'Expires': '-1'
    'Vary': 'Accept-Encoding,Accept-Encoding'
    'x-ms-request-id': '<x-ms-request-id>'
    'x-ms-ratelimit-remaining-subscription-reads': '11996'
    'Request-Context': 'appId=cid-v1:<some_id>'
    'x-ms-response-type': 'standard'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-<location>-01'
    'x-request-time': '0.161'
    'x-ms-correlation-request-id': '<some_id>'
    'x-ms-routing-request-id': 'CANADAEAST:20231219T174243Z:<some_id>'
    'Date': 'Tue, 19 Dec 2023 17:42:43 GMT'
Response content:
{
  "error": {
    "code": "DeploymentFailed",
    "message": "InferencingClient HttpRequest error, error detail: {\"errors\":{\"InferenceContainerConfiguration.ServingContainer.Image\":[\"'Image' must not be empty.\"]},\"type\":\"https://tools.ietf.org/html/rfc7231#section-6.5.1\",\"title\":\"One or more validation errors occurred.\",\"status\":400,\"traceId\":\"<trace_id>\"}",
    "details": [
      {
        "code": "DeploymentFailed",
        "message": "InferencingClient HttpRequest error, error detail: {\"errors\":{\"InferenceContainerConfiguration.ServingContainer.Image\":[\"'Image' must not be empty.\"]},\"type\":\"https://tools.ietf.org/html/rfc7231#section-6.5.1\",\"title\":\"One or more validation errors occurred.\",\"status\":400,\"traceId\":\"<trace_id>\"}",
        "details": [],
        "additionalInfo": []
      }
    ],
    "additionalInfo": [
      {
        "type": "MmsOperation",
        "info": {
          "id": "<some_id>",
          "operationType": "AmlArcService",
          "state": "Failed",
          "createdTime": "2023-12-19T17:42:37.2440026Z",
          "endTime": "2023-12-19T17:42:38.4024803Z",
          "error": {
            "code": "DeploymentFailed",
            "statusCode": 400,
            "message": "InferencingClient HttpRequest error, error detail: {\"errors\":{\"InferenceContainerConfiguration.ServingContainer.Image\":[\"'Image' must not be empty.\"]},\"type\":\"https://tools.ietf.org/html/rfc7231#section-6.5.1\",\"title\":\"One or more validation errors occurred.\",\"status\":400,\"traceId\":\"<trace_id>\"}"
          },
          "resourceLocation": "/api/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/services/<model_deployment_name>-<endpoint_name>",
          "operationLog": "https://<storage_account_id>.blob.core.windows.net/azureml/ImageLogs/<some_id>/build.log?sv=2019-07-07&sr=b&sig=aRNhAiV1Pzadw1nRe%2B%2FkM21nsXJz4lRe%2B5pn6EUsPKQ%3D&st=2023-12-19T17%3A37%3A43Z&se=2023-12-20T01%3A42%3A43Z&sp=r",
          "parentRequestId": "<some_id>",
          "streamingOperationLog": "20231219174237|Creating Container Registry if not exists."
        }
      }
    ]
  },
  "id": "/subscriptions/<subscription_id>/providers/Microsoft.MachineLearningServices/locations/<location>/mfeOperationsStatus/od:<some_id>:<some_id>",
  "name": "od:<some_id>:<some_id>",
  "status": "Failed",
  "startTime": "2023-12-19T17:42:37.2440026+00:00",
  "endTime": "2023-12-19T17:42:38.4024803+00:00",
  "percentComplete": 100.0
}
.
banibrata-de commented 7 months ago

Thanks for sharing the logs we are doing some more investigation !

siyuZL commented 7 months ago

We have conclusion in this issue #33611, the registry is not supported in Kubernetes compute and have no plan to support. Error is expected but error message should be improved.