[FR] Support pandas categorical during schema enforcement

timvink commented 1 month ago

Willingness to contribute

No. I cannot contribute this feature at this time.

Proposal Summary

I would like support for pandas categorical data when using mlflow signatures.

Using the category dtype is common when using boosting algorithms. Here's an example from the scikit-learn docs: Categorical Feature Support in Gradient Boosting.

Here's the current error message when trying to use the categorical dtype with mlflow:

import pandas as pd
from mlflow.models import infer_signature
from mlflow.pyfunc import _enforce_schema

df = pd.DataFrame({
    "col_1": ["1", "2", "3"],
    "col_2": ["1", "2", "3"],
})
df['col_2'] = df['col_2'].astype('category')

signature = infer_signature(df)
_enforce_schema(df, signature.inputs)
#> MlflowException: Incompatible input types for column col_2. Can not safely convert category to <U0.

Motivation

What is the use case for this feature?

Using boosting algorithms with categorical input data

Why is this use case valuable to support for MLflow users in general?

Using the category dtype is common when using boosting algorithms. Here's an example from the scikit-learn docs: Categorical Feature Support in Gradient Boosting.

Why is this use case valuable to support for your project(s) or organization?

We use boosting algorithms with categorical input data

Why is it currently difficult to achieve this use case?

It's not supported

Details

No response

What component(s) does this bug affect?

[ ] area/artifacts: Artifact stores and artifact logging
[ ] area/build: Build and test infrastructure for MLflow
[X] area/deployments: MLflow Deployments client APIs, server, and third-party Deployments integrations
[ ] area/docs: MLflow documentation pages
[ ] area/examples: Example code
[ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
[X] area/models: MLmodel format, model serialization/deserialization, flavors
[ ] area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
[ ] area/projects: MLproject format, project running backends
[ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
[ ] area/server-infra: MLflow Tracking server backend
[ ] area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

[ ] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
[ ] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
[ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
[ ] area/windows: Windows support

What language(s) does this bug affect?

[ ] language/r: R APIs and clients
[ ] language/java: Java APIs and clients
[ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

[ ] integrations/azure: Azure and Azure ML integrations
[ ] integrations/sagemaker: SageMaker integrations
[ ] integrations/databricks: Databricks integrations

timvink commented 1 month ago

github-actions[bot] commented 4 weeks ago

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.

timvink commented 2 weeks ago

This might actually also be classified as a bug, given the error message is Can not safely convert category to <U0. is not very helpful

mlflow / mlflow