If the job type is incorrect, change the 'type' property. with Azure CLI v2

[MislavSag]

Describe the bug If the job type is incorrect, change the 'type' property.

To Reproduce Steps to reproduce the behavior:

I have use the same structure as here: https://github.com/Azure/azureml-examples/tree/0849cbe797d1d524df9fe9d43ac8b36e75ea34ab/cli/jobs/train/r/accidents

When I run:

az ml job create -f .\job.yml --web

I get an error:

Command group 'ml job' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Warning: the provided input type data for 'OrderedDict([('data', OrderedDict([('local_path', 'data')])), ('mode', 'mount')])' is deprecated and will be removed in a future release. Use dataset instead.
Validation for CommandJobSchema failed:

 {
  "compute": [
    {
      "_schema": [
        "Value passed is not in set ['local']"
      ]
    },
    {
      "_schema": [
        "In order to specify an existing computes, please provide either of the following prefixed with 'azureml:':\n1. The full ARM ID for the resource, e.g.azureml:/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/computes/<resource_name>/<version-if applicable>)\n2. The short-hand name of the resource registered in the workspace, eg: azureml:<short-hand-name>:<version-if applicable>. For example, version 1 of the environment registered as 'my-env' in the workspace can be referenced as 'azureml:my-env:1'"
      ]
    },
    {
      "_schema": [
        "Not a valid string."
      ]
    }
  ],
  "environment": [
    {
      "docker": [
        "Unknown field."
      ]
    },
    {
      "_schema": [
        "In order to specify an existing environments, please provide either of the following prefixed with 'azureml:':\n1. The full ARM ID for the resource, e.g.azureml:/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace_name>/environments/<resource_name>/<version-if applicable>)\n2. The short-hand name of the resource registered in the workspace, eg: azureml:<short-hand-name>:<version-if applicable>. For example, version 1 of the environment registered as 'my-env' in the workspace can be referenced as 'azureml:my-env:1'"
      ]
    }
  ]
}

 If the job type is incorrect, change the 'type' property.

Before that I have tried hello world example and it worked.

Expected behavior I expect to get results in my ML service web window as with hello world.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

Issue Details

---

Describe the bug If the job type is incorrect, change the 'type' property. To Reproduce Steps to reproduce the behavior: I have use the same structure as here: https://github.com/Azure/azureml-examples/tree/0849cbe797d1d524df9fe9d43ac8b36e75ea34ab/cli/jobs/train/r/accidents When I run: ``` az ml job create -f .\job.yml --web ``` I get an error: ``` Command group 'ml job' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus Warning: the provided input type data for 'OrderedDict([('data', OrderedDict([('local_path', 'data')])), ('mode', 'mount')])' is deprecated and will be removed in a future release. Use dataset instead. Validation for CommandJobSchema failed: { "compute": [ { "_schema": [ "Value passed is not in set ['local']" ] }, { "_schema": [ "In order to specify an existing computes, please provide either of the following prefixed with 'azureml:':\n1. The full ARM ID for the resource, e.g.azureml:/subscriptions//resourceGroups//providers/Microsoft.MachineLearningServices/workspaces//computes//)\n2. The short-hand name of the resource registered in the workspace, eg: azureml::. For example, version 1 of the environment registered as 'my-env' in the workspace can be referenced as 'azureml:my-env:1'" ] }, { "_schema": [ "Not a valid string." ] } ], "environment": [ { "docker": [ "Unknown field." ] }, { "_schema": [ "In order to specify an existing environments, please provide either of the following prefixed with 'azureml:':\n1. The full ARM ID for the resource, e.g.azureml:/subscriptions//resourceGroups//providers/Microsoft.MachineLearningServices/workspaces//environments//)\n2. The short-hand name of the resource registered in the workspace, eg: azureml::. For example, version 1 of the environment registered as 'my-env' in the workspace can be referenced as 'azureml:my-env:1'" ] } ] } If the job type is incorrect, change the 'type' property. ``` Before that I have tried hello world example and it worked. Expected behavior I expect to get results in my ML service web window as with hello world.

Author:	MislavSag
Assignees:	-
Labels:	`Service Attention`, `Machine Learning`, `customer-reported`
Milestone:	-

[yonzhan]

route to service team

[luigiw]

The YAML file is out of date for the latest CLI version. I'll follow up on an update of that example.

[lostmygithubaccount]

Hello, the example (and YAML file) is up to date on the main branch of the repo. This issue is pointing to a commit from May 2021. There have been a number of breaking changes to the CLI (v2) while in preview since then, including some of the things you're seeing in this issue.

The update to date corresponding example for this which works on the recent extension versions is here: https://github.com/Azure/azureml-examples/tree/main/cli/jobs/single-step/r/accidents

[MislavSag]

@lostmygithubaccount ,

Thanks for sharing. In meantime, I have found another repo with structure that also worked. Your is again different, but hope it will work too.

Before closing the issue, I would like for advice on azure services. I know thi issue is not related to that, but maybe you can answer in one or two sentences. I would like to run my R code in cloud using Azure services. Before I have used Azure Batch for that, but for some reason repos for Azure Batch in R are archived (https://github.com/Azure/doAzureParallel and https://github.com/Azure/azure-batch-rscripts-manager). It seems to me Azure Batch is not planned to be used with auzre in the feature.

I have tried to find similar services on Azure, but I can't find it. Is this Azure ML with CLI v2 replacement for Azure batch in some sense? Azure functions are not appropiate since my script runs long time on many nodes. Other fundtions include Spark which I don't use.

[lostmygithubaccount]

Azure Machine Learning is, from the documentation overview, a product for:

Azure Machine Learning is for individuals and teams implementing MLOps within their organization to bring machine learning models into production in a secure and auditable production environment.

Data scientists and ML engineers will find tools to accelerate and automate their day-to-day workflows. Application developers will find tools for integrating models into applications or services. Platform developers will find a robust set of tools, backed by durable Azure Resource Manager APIs, for building advanced ML tooling.

Enterprises working in the Microsoft Azure cloud will find familiar security and role-based access control (RBAC) for infrastructure. You can set up a project to deny access to protected data and select operations.

I am not personally very familiar with Azure Batch as a product or its intended use cases, and I do want to be clear that Azure ML is targeting automation, teams of ML professionals, and other ML-oriented features, but regardless I think it's fair to view the v2 CLI extension as a replacement for Azure Batch, likely with some modification needed to migrate. Currently v2 supports arbitrary command jobs and can be distributed with tensorflow, pytorch, and mpi. These also allow for more flexible distribution e.g. with Dask or Spark, though support there is relatively limited.

[MislavSag]

Yes, it seems to me Azure ML support much more than just ML. It supports running any kind of code. For example I can run R script with as many nodes in parallel as I want. This is exactly what Azure batch was doing. But it supports lot more options I man not interesed in. I am not sure if the service is more expensive because of this additional options.

Can I use cheap shared nodes in azure VM?

[MislavSag]

I have found the option while creating the compute which allow me to use shared nodes.