[FR] Add class_probability to the input "eval_df" of "custom_metrics" function in recipes

[e-taghizadeh]

Willingness to contribute

Yes. I would be willing to contribute this feature with guidance from the MLflow community.

Proposal Summary

For developing some metrics, one would need to use the output probability. I propose to add class_probability to the input "eval_df" of "custom_metrics" function.

Motivation

What is the use case for this feature?

For some classification algorithms, I'd like to use precision_recall_auc as the primary metric however I cannot implement it, as it requires the output probability.

Why is this use case valuable to support for MLflow users in general?

In many imbalanced datasets, these metrics are the suggested metrics in the community.

Some Thoughts

For the estimators that do not support "predict_prob" these columns could be removed or set to None in the "eval_df" DF. Currently, the precision_recall_auc or roc_auc values are automatically computed in the "evaluation" step if the estimator has some specifications, a similar approach can be used in the recipes.

Details

No response

What component(s) does this bug affect?

• [ ] area/artifacts: Artifact stores and artifact logging
• [ ] area/build: Build and test infrastructure for MLflow
• [ ] area/docs: MLflow documentation pages
• [ ] area/examples: Example code
• [ ] area/gateway: AI Gateway service, Gateway client APIs, third-party Gateway integrations
• [ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• [ ] area/models: MLmodel format, model serialization/deserialization, flavors
• [X] area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• [ ] area/projects: MLproject format, project running backends
• [ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• [ ] area/server-infra: MLflow Tracking server backend
• [ ] area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• [ ] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• [ ] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• [ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• [ ] area/windows: Windows support

What language(s) does this bug affect?

• [ ] language/r: R APIs and clients
• [ ] language/java: Java APIs and clients
• [ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

• [ ] integrations/azure: Azure and Azure ML integrations
• [ ] integrations/sagemaker: SageMaker integrations
• [ ] integrations/databricks: Databricks integrations

[github-actions[bot]]

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.