databricks / mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.
https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html
Apache License 2.0
416 stars 141 forks source link

Model validation for FS models #165

Closed aliazzzdat closed 1 month ago

aliazzzdat commented 1 month ago

FS models cannot predict as pyfunc models yet (issue #70), thus we can't use MLflow Evaluate with FS models.

However, MLflow Evaluate can take a function instead of a pyfunc model_uri. This means we can call an FS model in the function and use this function in the Evaluate method.

Although this is a good solution, we still cannot enable baseline comparison, as we need to provide a model_uri of a pyfunc model (so this means it's not possible to compare a new FS model with a baseline FS model).

arpitjasa-db commented 1 month ago

@aliazzzdat thanks for all the changes! Looks like some tests are failing, can we fix those and also do a full instantiation of Stacks to confirm everything is still working as expected? Version changes can be tricky, and want to make sure we don't break anything

aliazzzdat commented 1 month ago

Featurization pipeline works on DBR 15.3 and with MLflow 2.11

Screenshot 2024-07-19 at 21 43 00

Model training pipeline works on DBR 15.3 and with MLflow 2.11

Screenshot 2024-07-19 at 21 43 29

Validation works with a Feature Store model

Screenshot 2024-07-19 at 21 43 57 Screenshot 2024-07-19 at 21 44 16 Screenshot 2024-07-19 at 21 44 30
aliazzzdat commented 1 month ago

The batch inference works as expected once we create the feature_store_inference_input table

Screenshot 2024-07-19 at 21 54 34