search

Azure / azure-cli-extensions

Public Repository for Extensions of Azure CLI.
https://docs.microsoft.com/en-us/cli/azure
MIT License
381 stars 1.2k forks source link

Azure ML extension does not provide sufficient information in error message #6662

Open mattmazzola opened 1 year ago

mattmazzola commented 1 year ago

Describe the bug

I try to submit a job to Azure ML Workspace using

az ml job create -f Elevater_Toolkit_IC/azcli_configs/elevater_example.yaml

At one point I had been able to submit jobs using this same command successfully and now I cannot. I tried to remove all potential problems from the YAML config file to so now it ONLY attempt to print "Hello, World", but yet it still fails.

Error

(UserError) System.Net.Http.HttpConnectionResponseContent
Code: UserError
Message: System.Net.Http.HttpConnectionResponseContent

This is problem. The message is too generic

Related command

az ml job create -f Elevater_Toolkit_IC/azcli_configs/elevater_example.yaml

Errors

(biomed) vscode ➜ /workspaces/biomed_clip (mattm/az-cli-v2) $ az ml job create -f Elevater_Toolkit_IC/azcli_configs/elevater_example.yaml          

Class AutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class AutoDeleteConditionSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseAutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class IntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class ProtectionLevelSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseIntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
(UserError) System.Net.Http.HttpConnectionResponseContent
Code: UserError
Message: System.Net.Http.HttpConnectionResponseContent

Issue script & Debug output

(biomed) vscode ➜ /workspaces/biomed_clip (mattm/az-cli-v2) $ az ml job create -f Elevater_Toolkit_IC/azcli_configs/elevater_example.yaml --debug

cli.knack.cli: Command arguments: ['ml', 'job', 'create', '-f', 'Elevater_Toolkit_IC/azcli_configs/elevater_example.yaml', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f478ced9480>, <function OutputProducer.on_global_arguments at 0x7f478ce303a0>, <function CLIQuery.on_global_arguments at 0x7f478cc4d5a0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'ml': ['azext_mlv2']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
azext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.
cli.azure.cli.core: ml                        2.305        20       139  /home/vscode/.azure/cliextensions/ml
cli.azure.cli.core: Total (1)                 2.305        20       139  
cli.azure.cli.core: Loaded 20 groups, 139 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : ml job create
cli.azure.cli.core: Command table: ml job create
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f478c01edd0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/vscode/.azure/commands/2023-08-18.22-25-40.ml_job_create.7660.log'.
az_command_data_logger: command args: ml job create -f {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f478c037910>]
cli.knack.commands: Configured default 'hanover-dev' for arg resource_group_name
cli.knack.commands: Configured default 'naoto-wus2' for arg workspace_name
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f478c0a1750>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f478c0a1870>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f478ce30430>, <function CLIQuery.handle_query_parameter at 0x7f478cc4d630>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f478c0a17e0>]
az_command_data_logger: extension name: ml
az_command_data_logger: extension version: 2.19.1
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=AzureMachineLearningWorkspaces
cli.azure.cli.core.auth.persistence: build_persistence: location='/home/vscode/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /home/vscode/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/kerberos', 'tenant_region_scope': 'WW', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=_ml_client_cli
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/kerberos', 'tenant_region_scope': 'WW', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
Class AutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class AutoDeleteConditionSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseAutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class IntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class ProtectionLevelSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseIntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
attr_dict: getting schema_ignored
attr_dict: setting schema_ignored to {}
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.azure.com/.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.azure.com/.default',), claims=None, kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: 48ef2000-9c01-4b6f-9d33-5f296015c2a8
azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/fb4af94d-4746-40f5-b8a4-11638ef18e25/resourceGroups/hanover-dev/providers/Microsoft.MachineLearningServices/workspaces/naoto-wus2/codes/8c9b99c6-50d4-44de-bbbb-226ced204ed1/versions?api-version=2023-04-01&hash=d8c6055850b1112432a985225e7c908c28c7f7965b725b2255ba089976f2d98c&hashVersion=202208'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '289f495a-3e16-11ee-8086-0242ac110003'
    'User-Agent': 'azureml-cli-v2/2.19.1 azure-ai-ml/1.9.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.10.10 (Linux-5.15.90.4-microsoft-standard-WSL2-x86_64-with-glibc2.31)'
    'Authorization': 'Bearer <removed>'
Request body:
This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/fb4af94d-4746-40f5-b8a4-11638ef18e25/resourceGroups/hanover-dev/providers/Microsoft.MachineLearningServices/workspaces/naoto-wus2/codes/8c9b99c6-50d4-44de-bbbb-226ced204ed1/versions?api-version=2023-04-01&hash=d8c6055850b1112432a985225e7c908c28c7f7965b725b2255ba089976f2d98c&hashVersion=202208 HTTP/1.1" 200 1123
azure.core.pipeline.policies._universal: Response status: '200'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Content-Length': '1123'
    'Content-Type': 'application/json; charset=utf-8'
    'Expires': '-1'
    'Vary': 'Accept-Encoding'
    'x-ms-request-id': '37817edb-fb24-4e26-89d1-9d12db38e84f'
    'x-ms-ratelimit-remaining-subscription-reads': '11999'
    'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d'
    'x-ms-response-type': 'standard'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-westus2-02'
    'x-request-time': '0.097'
    'x-ms-correlation-request-id': '37817edb-fb24-4e26-89d1-9d12db38e84f'
    'x-ms-routing-request-id': 'WESTUS2:20230818T222542Z:37817edb-fb24-4e26-89d1-9d12db38e84f'
    'X-Cache': 'CONFIG_NOCACHE'
    'X-MSEdge-Ref': 'Ref A: F6A7DD3D23694119A2C96DBE6213DE8A Ref B: CO6AA3150218025 Ref C: 2023-08-18T22:25:41Z'
    'Date': 'Fri, 18 Aug 2023 22:25:41 GMT'
Response content:
{
  "value": [
    {
      "id": "/subscriptions/fb4af94d-4746-40f5-b8a4-11638ef18e25/resourceGroups/hanover-dev/providers/Microsoft.MachineLearningServices/workspaces/naoto-wus2/codes/b975179b-2680-4483-be82-409271fd9bb2/versions/1",
      "name": "1",
      "type": "Microsoft.MachineLearningServices/workspaces/codes/versions",
      "properties": {
        "description": null,
        "tags": {},
        "properties": {
          "hash_sha256": "d8c6055850b1112432a985225e7c908c28c7f7965b725b2255ba089976f2d98c",
          "hash_version": "202208"
        },
        "isArchived": false,
        "isAnonymous": false,
        "codeUri": "https://naotowus22241333556.blob.core.windows.net:443/799d5e80-e-9790b233-7227-58b9-b379-f3df37cfb48a/azcli_configs",
        "provisioningState": "Succeeded"
      },
      "systemData": {
        "createdAt": "2023-08-18T21:53:03.0292962+00:00",
        "createdBy": "Matt Mazzola",
        "createdByType": "User",
        "lastModifiedAt": "2023-08-18T21:53:03.0292962+00:00",
        "lastModifiedBy": "Matt Mazzola",
        "lastModifiedByType": "User"
      }
    }
  ]
}
azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/fb4af94d-4746-40f5-b8a4-11638ef18e25/resourceGroups/hanover-dev/providers/Microsoft.MachineLearningServices/workspaces/naoto-wus2/codes/b975179b-2680-4483-be82-409271fd9bb2/versions/1?api-version=2023-04-01'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '289f495a-3e16-11ee-8086-0242ac110003'
    'User-Agent': 'azureml-cli-v2/2.19.1 azure-ai-ml/1.9.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.10.10 (Linux-5.15.90.4-microsoft-standard-WSL2-x86_64-with-glibc2.31)'
    'Authorization': 'Bearer <removed>'
Request body:
This request has no body
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/fb4af94d-4746-40f5-b8a4-11638ef18e25/resourceGroups/hanover-dev/providers/Microsoft.MachineLearningServices/workspaces/naoto-wus2/codes/b975179b-2680-4483-be82-409271fd9bb2/versions/1?api-version=2023-04-01 HTTP/1.1" 200 1002
azure.core.pipeline.policies._universal: Response status: '200'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Content-Length': '1002'
    'Content-Type': 'application/json; charset=utf-8'
    'Expires': '-1'
    'Vary': 'Accept-Encoding'
    'x-ms-request-id': 'c8ff3672-3fd0-4770-bd9d-e03b9fdc7bfd'
    'x-ms-ratelimit-remaining-subscription-reads': '11998'
    'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d'
    'x-ms-response-type': 'standard'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-westus2-02'
    'x-request-time': '0.134'
    'x-ms-correlation-request-id': 'c8ff3672-3fd0-4770-bd9d-e03b9fdc7bfd'
    'x-ms-routing-request-id': 'WESTUS2:20230818T222542Z:c8ff3672-3fd0-4770-bd9d-e03b9fdc7bfd'
    'X-Cache': 'CONFIG_NOCACHE'
    'X-MSEdge-Ref': 'Ref A: C505DF4054184C3FB466D4E4AAF264EB Ref B: CO6AA3150218025 Ref C: 2023-08-18T22:25:42Z'
    'Date': 'Fri, 18 Aug 2023 22:25:41 GMT'
Response content:
{
  "id": "/subscriptions/fb4af94d-4746-40f5-b8a4-11638ef18e25/resourceGroups/hanover-dev/providers/Microsoft.MachineLearningServices/workspaces/naoto-wus2/codes/b975179b-2680-4483-be82-409271fd9bb2/versions/1",
  "name": "1",
  "type": "Microsoft.MachineLearningServices/workspaces/codes/versions",
  "properties": {
    "description": null,
    "tags": {},
    "properties": {
      "hash_sha256": "d8c6055850b1112432a985225e7c908c28c7f7965b725b2255ba089976f2d98c",
      "hash_version": "202208"
    },
    "isArchived": false,
    "isAnonymous": false,
    "codeUri": "https://naotowus22241333556.blob.core.windows.net:443/799d5e80-e-9790b233-7227-58b9-b379-f3df37cfb48a/azcli_configs",
    "provisioningState": "Succeeded"
  },
  "systemData": {
    "createdAt": "2023-08-18T21:53:03.0292962+00:00",
    "createdBy": "Matt Mazzola",
    "createdByType": "User",
    "lastModifiedAt": "2023-08-18T21:53:03.0292962+00:00",
    "lastModifiedBy": "Matt Mazzola",
    "lastModifiedByType": "User"
  }
}
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.azure.com/.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.azure.com/.default',), claims=None, kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: 5309aa02-d26b-4c7a-b249-8fb602c032e2
azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/fb4af94d-4746-40f5-b8a4-11638ef18e25/resourceGroups/hanover-dev/providers/Microsoft.MachineLearningServices/workspaces/naoto-wus2/environments/openmpi4.1.0-cuda11.6-cudnn8-ubuntu20.04/versions?api-version=2023-04-01-preview&$orderBy=createdtime%20desc&$top=1'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': '289f495a-3e16-11ee-8086-0242ac110003'
    'User-Agent': 'azureml-cli-v2/2.19.1 azure-ai-ml/1.9.0 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.10.10 (Linux-5.15.90.4-microsoft-standard-WSL2-x86_64-with-glibc2.31)'
    'Authorization': 'Bearer <removed>'
Request body:
This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/fb4af94d-4746-40f5-b8a4-11638ef18e25/resourceGroups/hanover-dev/providers/Microsoft.MachineLearningServices/workspaces/naoto-wus2/environments/openmpi4.1.0-cuda11.6-cudnn8-ubuntu20.04/versions?api-version=2023-04-01-preview&$orderBy=createdtime%20desc&$top=1 HTTP/1.1" 404 818
azure.core.pipeline.policies._universal: Response status: '404'
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Content-Length': '818'
    'Content-Type': 'application/json; charset=utf-8'
    'Expires': '-1'
    'Vary': 'Accept-Encoding'
    'x-ms-request-id': 'b2e83e59-9a03-489e-bd57-b857dec3fe3b'
    'x-ms-ratelimit-remaining-subscription-reads': '11999'
    'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d'
    'x-ms-response-type': 'error'
    'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
    'X-Content-Type-Options': 'nosniff'
    'x-aml-cluster': 'vienna-westus2-01'
    'x-request-time': '0.120'
    'x-ms-correlation-request-id': 'b2e83e59-9a03-489e-bd57-b857dec3fe3b'
    'x-ms-routing-request-id': 'WESTUS2:20230818T222542Z:b2e83e59-9a03-489e-bd57-b857dec3fe3b'
    'X-Cache': 'CONFIG_NOCACHE'
    'X-MSEdge-Ref': 'Ref A: 9554B7A5B74048BAAB2FFC6DCC1A4B1E Ref B: CO6AA3150220037 Ref C: 2023-08-18T22:25:42Z'
    'Date': 'Fri, 18 Aug 2023 22:25:42 GMT'
Response content:
{
  "error": {
    "code": "UserError",
    "message": "System.Net.Http.HttpConnectionResponseContent",
    "details": [],
    "additionalInfo": [
      {
        "type": "ComponentName",
        "info": {
          "value": "managementfrontend"
        }
      },
      {
        "type": "Correlation",
        "info": {
          "value": {
            "operation": "7f744fd76294484c3e94c4f3759d86fa",
            "request": "67792ab76a61964b"
          }
        }
      },
      {
        "type": "Environment",
        "info": {
          "value": "westus2"
        }
      },
      {
        "type": "Location",
        "info": {
          "value": "westus2"
        }
      },
      {
        "type": "Time",
        "info": {
          "value": "2023-08-18T22:25:42.6567049+00:00"
        }
      }
    ]
  }
}
Traceback (most recent call last):
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create
    job = ml_client.jobs.create_or_update(job=job, skip_validation=skip_validation)
  File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 337, in wrapper
    return_value = f(*args, **kwargs)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 610, in create_or_update
    raise ex
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 542, in create_or_update
    self._resolve_arm_id_or_upload_dependencies(job)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 894, in _resolve_arm_id_or_upload_dependencies
    self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1108, in _resolve_arm_id_or_azureml_id
    job = self._resolve_arm_id_for_command_job(job, resolver)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1149, in _resolve_arm_id_for_command_job
    job.environment = resolver(job.environment, azureml_type=AzureMLResourceType.ENVIRONMENT)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 178, in get_asset_arm_id
    name, version = self._resolve_name_version_from_name_label(asset, azureml_type)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 426, in _resolve_name_version_from_name_label
    _resolve_label_to_asset(
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 949, in _resolve_label_to_asset
    return resolver(name)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_environment_operations.py", line 371, in _get_latest_version
    result = _get_latest(
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 799, in _get_latest
    latest = result.next()
  File "/opt/az/lib/python3.10/site-packages/azure/core/paging.py", line 128, in __next__
    return next(self._page_iterator)
  File "/opt/az/lib/python3.10/site-packages/azure/core/paging.py", line 76, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2023_04_01_preview/operations/_environment_versions_operations.py", line 333, in get_next
    map_error(status_code=response.status_code, response=response, error_map=error_map)
  File "/opt/az/lib/python3.10/site-packages/azure/core/exceptions.py", line 107, in map_error
    raise error
azure.core.exceptions.ResourceNotFoundError: (UserError) System.Net.Http.HttpConnectionResponseContent
Code: UserError
Message: System.Net.Http.HttpConnectionResponseContent
cli: None
cli: Received HttpResponseError: Traceback (most recent call last):
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create
    job = ml_client.jobs.create_or_update(job=job, skip_validation=skip_validation)
  File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 337, in wrapper
    return_value = f(*args, **kwargs)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 610, in create_or_update
    raise ex
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 542, in create_or_update
    self._resolve_arm_id_or_upload_dependencies(job)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 894, in _resolve_arm_id_or_upload_dependencies
    self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1108, in _resolve_arm_id_or_azureml_id
    job = self._resolve_arm_id_for_command_job(job, resolver)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1149, in _resolve_arm_id_for_command_job
    job.environment = resolver(job.environment, azureml_type=AzureMLResourceType.ENVIRONMENT)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 178, in get_asset_arm_id
    name, version = self._resolve_name_version_from_name_label(asset, azureml_type)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 426, in _resolve_name_version_from_name_label
    _resolve_label_to_asset(
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 949, in _resolve_label_to_asset
    return resolver(name)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_environment_operations.py", line 371, in _get_latest_version
    result = _get_latest(
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 799, in _get_latest
    latest = result.next()
  File "/opt/az/lib/python3.10/site-packages/azure/core/paging.py", line 128, in __next__
    return next(self._page_iterator)
  File "/opt/az/lib/python3.10/site-packages/azure/core/paging.py", line 76, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2023_04_01_preview/operations/_environment_versions_operations.py", line 333, in get_next
    map_error(status_code=response.status_code, response=response, error_map=error_map)
  File "/opt/az/lib/python3.10/site-packages/azure/core/exceptions.py", line 107, in map_error
    raise error
azure.core.exceptions.ResourceNotFoundError: (UserError) System.Net.Http.HttpConnectionResponseContent
Code: UserError
Message: System.Net.Http.HttpConnectionResponseContent

cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/custom/job.py", line 104, in ml_job_create
    log_and_raise_error(err, debug, yaml_operation=yaml_operation)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 181, in log_and_raise_error
    raise cli_error
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/custom/job.py", line 90, in ml_job_create
    job = ml_client.jobs.create_or_update(job=job, skip_validation=skip_validation)
  File "/opt/az/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_telemetry/activity.py", line 337, in wrapper
    return_value = f(*args, **kwargs)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 610, in create_or_update
    raise ex
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 542, in create_or_update
    self._resolve_arm_id_or_upload_dependencies(job)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 894, in _resolve_arm_id_or_upload_dependencies
    self._resolve_arm_id_or_azureml_id(job, self._orchestrators.get_asset_arm_id)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1108, in _resolve_arm_id_or_azureml_id
    job = self._resolve_arm_id_for_command_job(job, resolver)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_job_operations.py", line 1149, in _resolve_arm_id_for_command_job
    job.environment = resolver(job.environment, azureml_type=AzureMLResourceType.ENVIRONMENT)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 178, in get_asset_arm_id
    name, version = self._resolve_name_version_from_name_label(asset, azureml_type)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_operation_orchestrator.py", line 426, in _resolve_name_version_from_name_label
    _resolve_label_to_asset(
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 949, in _resolve_label_to_asset
    return resolver(name)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/operations/_environment_operations.py", line 371, in _get_latest_version
    result = _get_latest(
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_utils/_asset_utils.py", line 799, in _get_latest
    latest = result.next()
  File "/opt/az/lib/python3.10/site-packages/azure/core/paging.py", line 128, in __next__
    return next(self._page_iterator)
  File "/opt/az/lib/python3.10/site-packages/azure/core/paging.py", line 76, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/home/vscode/.azure/cliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/_restclient/v2023_04_01_preview/operations/_environment_versions_operations.py", line 333, in get_next
    map_error(status_code=response.status_code, response=response, error_map=error_map)
  File "/opt/az/lib/python3.10/site-packages/azure/core/exceptions.py", line 107, in map_error
    raise error
azure.core.exceptions.ResourceNotFoundError: (UserError) System.Net.Http.HttpConnectionResponseContent
Code: UserError
Message: System.Net.Http.HttpConnectionResponseContent

cli.azure.cli.core.azclierror: (UserError) System.Net.Http.HttpConnectionResponseContent
Code: UserError
Message: System.Net.Http.HttpConnectionResponseContent
az_command_data_logger: (UserError) System.Net.Http.HttpConnectionResponseContent
Code: UserError
Message: System.Net.Http.HttpConnectionResponseContent
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f478c01f010>]
az_command_data_logger: exit code: 3
cli.__main__: Command ran in 4.302 seconds (init: 0.284, invoke: 4.019)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3976 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/bin/../../opt/az/bin/python3 /opt/az/lib/python3.10/site-packages/azure/cli/telemetry/__init__.py /home/vscode/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

The error message should tell me precisely what is wrong so I can make changes and fix it.

From above it looks like a 404 Not Found from attempting to find the workspace but the workspace does exist.

Environment Summary

Linux-5.15.90.4-microsoft-standard-WSL2-x86_64-with-glibc2.31, Ubuntu 20.04.6 LTS Python 3.10.10 Installer: DEB

azure-cli 2.51.0

Extensions: ml 2.19.1

Dependencies: msal 1.24.0b1 azure-mgmt-resource 23.1.0b2

Additional context

No response

yonzhan commented 1 year ago

Thank you for opening this issue, we will look into it.

mattmazzola commented 1 year ago

From experimenting more I found that it was because of the syntax used for the environment configuration.

Error Causing Syntax

environment: azureml:openmpi4.1.0-cuda11.6-cudnn8-ubuntu20.04@latest

Correct Syntax

environment:
  image: azureml/openmpi4.1.0-cuda11.6-cudnn8-ubuntu20.04:latest

I think you should be able to give a better error message. The debug output has

https://management.azure.com:443 "GET /subscriptions/fb4af94d-4746-40f5-b8a4-11638ef18e25/resourceGroups/hanover-dev/providers/Microsoft.MachineLearningServices/workspaces/naoto-wus2/environments/openmpi4.1.0-cuda11.6-cudnn8-ubuntu20.04/versions?api-version=2023-04-01-preview&$orderBy=createdtime%20desc&$top=1

This includes the request that failed

I would expect it to say something like

Your configuration file attempted to use the environment "openmpi4.1.0-cuda11.6-cudnn8-ubuntu20.04", but that environment does not exist. Available environments are [a,b,c,d....'. Please see environments here: https://www....

navba-MSFT commented 1 year ago

Non customer-reported. Adding Service team to look into this.

microsoft-github-policy-service[bot] commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

microsoft-github-policy-service[bot] commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

matsuobasho commented 11 months ago

I get this same vague and useless error message when I specify a non-existing environment:

image

Please address this, it's been almost 2 months since the issue has been brought to the team's attention. These types of bugs are a major deterrent to practical use of AzureML cli.

bo-josh-chen commented 5 months ago

Hi team, I'm trying to submit and create an AML job using Azure CLI and yml file on a workspace, I got the same error message but unfortunately, with this message, I couldn't get enough information to debug it, any update would be appreciated. image