search

databricks / mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.
https://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.html
Apache License 2.0
425 stars 143 forks source link

Model validation with feature store should be supported. #70

Closed mingyu89 closed 1 month ago

mingyu89 commented 1 year ago

Mode validation uses mlflow.evaluate. Currently it doesn't support evaluating models registered with databricks feature store.

aminebizid commented 1 year ago

Any news about this limitation please?

jackroseman commented 1 year ago

I am also wondering when this feature is coming. Thanks!

skaliy commented 9 months ago

Any updates? :)

arpitjasa-db commented 8 months ago

Hi we're looking into adding support for this, the tricky part is we need to update underlying feature store and MLflow components to get this to work, but will hopefully have an update for you all soon!

david-straub commented 1 month ago

Any updates?

arpitjasa-db commented 1 month ago

@david-straub we are currently reviewing a PR to add this functionality, but note this workaround will be slightly limited in that we cannot enable baseline comparison, as we need to provide a model_uri of a pyfunc model (so this means it's not possible to compare a new FS model with a baseline FS model)

arpitjasa-db commented 1 month ago

We just merged https://github.com/databricks/mlops-stacks/pull/165 thanks to @aliazzzdat! This workaround should resolve most use cases for this issue, but if there's any other problems, please open another issue thank you!