Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.55k stars 2.77k forks source link

Unable to create or update run <AUTOML Run ID> : Orchestrator can not be modified after a value is set.. #29780

Open aayushsin opened 1 year ago

aayushsin commented 1 year ago

Describe the bug I am trying to train a simple classification model using https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-machine-learning-tutorial. It runs fine while using the local computer cluster. I created a new compute cluster and there it is giving an error.

To Reproduce Steps to reproduce the behavior: `from azureml.core.compute import AmlCompute, ComputeTarget

compute_name = 'testCluster' if not compute_name in ws.compute_targets : print('creating a new compute target...') provisioning_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2', min_nodes=0, max_nodes=1) compute_target = ComputeTarget.create(ws, compute_name, provisioning_config)

compute_target.wait_for_completion(
    show_output=True, min_node_count=None, timeout_in_minutes=20)

print(compute_target.get_status().serialize())

compute_target = ws.compute_targets[compute_name]`

`import logging

automl_settings = { "iteration_timeout_minutes": 5, "experiment_timeout_minutes": 30, "enable_early_stopping": True, "primary_metric": 'accuracy', "featurization": 'auto', "verbosity": logging.INFO, "n_cross_validations": 2 }`

`from azureml.train.automl import AutoMLConfig

automl_config = AutoMLConfig(task='classification', debug_log='automated_ml_errors.log', training_data = dataset_training, spark_context = sc, model_explainability = False, compute_target=compute_target, label_column_name ="label",**automl_settings)`

`from azureml.core.experiment import Experiment

experiment = Experiment(ws, "aml-synapse-classification") tags = {"Synapse": "classification"} local_run = experiment.submit(automl_config, show_output=True, tags = tags)

run_details = local_run.get_details()`

Expected behavior The training should be a success

Error ITERATION PIPELINE DURATION METRIC BEST Received unrecognized parameter dataset_id 0 0:00:00 nan nan ERROR: { "additional_properties": {}, "error": { "additional_properties": { "debugInfo": null }, "code": "UserError", "severity": null, "message": "Unable to create or update run AutoML_bed2e2ac-38df-4b14-b960-546d9228eff2_0 : Orchestrator can not be modified after a value is set..", "message_format": "Unable to create or update run {RunId} : {ReasonPhrase}.", "message_parameters": { "RunId": "AutoML_bed2e2ac-38df-4b14-b960-546d9228eff2_0", "ReasonPhrase": "Orchestrator can not be modified after a value is set." }, "reference_code": null, "details_uri": null, "target": "AddOrModifyRun", "details": [], "inner_error": { "additional_properties": {}, "code": "BadArgument", "inner_error": { "additional_properties": {}, "code": "CreatRunBadRequest", "inner_error": null } }, "additional_info": null }, "correlation": { "operation": "1e983e83e81559e76a6879184a1d903f", "request": "14e7446f60d74d82" }, "environment": "centralus", "location": "centralus", "time": {}, "component_name": "execution" }

Additional context Add any other context about the problem here.

ghost commented 1 year ago

Thank you for your feedback. This has been routed to the support team for assistance.

github-actions[bot] commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github @Azure/azure-ml-sdk.