Azure / MachineLearningNotebooks

Python notebooks with ML and deep learning examples with Azure Machine Learning Python SDK | Microsoft
https://docs.microsoft.com/azure/machine-learning/service/
MIT License
4k stars 2.49k forks source link

AutoML experiment: get model and metrics for any algorithm (not only for the best one) #1885

Open movingabout opened 1 year ago

movingabout commented 1 year ago

What I'm trying to do

For an AutoML Forecasting experiment, I'd like to compare the performance of the best model with the performance of another model from the same experiment.

For an AutoML run, I understand how to get the best performing model and its metrics like this:

# ...initialize MLFlow client...
mlflow_parent_run = mlflow_client.get_run('upbeat_square_abs3942')
best_child_run_id = mlflow_parent_run.data.tags["automl_best_child_run_id"]
best_run = mlflow_client.get_run(best_child_run_id)
best_run.data.metrics
# etc...

But how can I fetch the job for any model based on the algorithm name? Something like:

# pseudocode:
mlflow_client.get_automl_run_by_algorithm('XGBoostRegressor')

So far, I managed to figure out the following:

  1. list of algorithms used in the AutoML experiment

    mlflow_parent_run.data.tags['pipeline_id_000']
    #  '__AutoML_Naive__;__AutoML_SeasonalNaive__;__AutoML_Average__;__AutoML_SeasonalAverage__;__AutoML_Ensemble__'

    However, this list seems to be in an arbitrary order and I struggle to get the corresponding job names for the algorithms.

  2. "internal" job names for the child runs

The child runs seem to have different names than the names shown in Azure ML Studio. They are named for instance upbeat_square_abs3942_2 - i.e. the name of the parent run upbeat_square_abs3942 followed by underscore plus a number (_2in this example).

But Azure ML Studio displays names like (no upbeat_square_abs3942_2 to be found):

image

So this code works:

child_run = mlflow_client.get_run('upbeat_square_abs3942_2')

but using a name shown in the screenshot above throws an exception, e.g.

child_run = mlflow_client.get_run('green_floor_0ln3tlpv')

Question

How can I obtain the model and metrics for any algorithm used in the experiment?

Thanks!