[X] I have read and agree to submit bug reports in accordance with the issues policy
Where did you encounter this bug?
Local machine
Willingness to contribute
Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
MLflow version
poetry show mlflow-skinny output:
name : mlflow-skinny
version : 2.8.0
description : MLflow: A Platform for ML Development and Productionization
dependencies
- click >=7.0,<9
- cloudpickle <3
- databricks-cli >=0.8.7,<1
- entrypoints <1
- gitpython >=2.1.0,<4
- importlib-metadata >=3.7.0,<4.7.0 || >4.7.0,<7
- packaging <24
- protobuf >=3.12.0,<5
- pytz <2024
- pyyaml >=5.1,<7
- requests >=2.17.3,<3
- sqlparse >=0.4.0,<1
System information
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): WSL, Ubuntu 22.04.3 LTS on Windows 10 x86_64
Python version: 3.10.12
yarn version, if running the dev UI: N/A
Describe the problem
Using an mlflow client with a tracking URI pointing to a named profile and the registry UI any call that connects to the workspace fails with an authentication error.
I have the env variable DATABRICKS_CONFIG_FILE set, but not DATABRICKS_HOST or DATABRICKS_TOKEN
import mlflow
client = mlflow.MlflowClient(tracking_uri="databricks://<my workspace>", registry_uri="databricks-uc")
client.search_registered_models() # All calls will fail here, this is just an example.
this gives the following output
Traceback (most recent call last):
File "/home/rohan/Documents/dk_datasci_model_pipeline/scripts/check_mlflow_connection.py", line 11, in <module>
client.search_registered_models()
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/tracking/client.py", line 2372, in search_registered_models
return self._get_registry_client().search_registered_models(
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/tracking/_model_registry/client.py", line 115, in search_registered_models
return self.store.search_registered_models(filter_string, max_results, order_by, page_token)
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/store/_unity_catalog/registry/rest_store.py", line 300, in search_registered_models
response_proto = self._call_endpoint(SearchRegisteredModelsRequest, req_body)
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/store/model_registry/base_rest_store.py", line 43, in _call_endpoint
self.get_host_creds(), endpoint, method, json_body, response_proto, extra_headers
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/utils/databricks_utils.py", line 439, in get_databricks_host_creds
config = provider.get_config()
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/databricks_cli/configure/provider.py", line 134, in get_config
raise InvalidConfigurationError.for_profile(None)
databricks_cli.utils.InvalidConfigurationError: You haven't configured the CLI yet! Please configure by entering `/home/rohan/Documents/dk_datasci_model_pipeline/scripts/check_mlflow_connection.py configure`
My config file has a section for the given workspace, and the credentials are valid.
If I adjust the authentication to set DATABRICKS_HOST and DATABRICKS_TOKEN instead, everything works fine
import mlflow
client = mlflow.MlflowClient(tracking_uri="databricks://<my workspace>", registry_uri="databricks-uc")
client.search_registered_models() # All calls will fail here, this is just an example.
Stack trace
Traceback (most recent call last):
File "/home/rohan/Documents/dk_datasci_model_pipeline/scripts/check_mlflow_connection.py", line 11, in <module>
client.search_registered_models()
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/tracking/client.py", line 2372, in search_registered_models
return self._get_registry_client().search_registered_models(
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/tracking/_model_registry/client.py", line 115, in search_registered_models
return self.store.search_registered_models(filter_string, max_results, order_by, page_token)
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/store/_unity_catalog/registry/rest_store.py", line 300, in search_registered_models
response_proto = self._call_endpoint(SearchRegisteredModelsRequest, req_body)
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/store/model_registry/base_rest_store.py", line 43, in _call_endpoint
self.get_host_creds(), endpoint, method, json_body, response_proto, extra_headers
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/mlflow/utils/databricks_utils.py", line 439, in get_databricks_host_creds
config = provider.get_config()
File "/home/rohan/.cache/pypoetry/virtualenvs/dk-datasci-model-pipeline-fXF56sVU-py3.10/lib/python3.10/site-packages/databricks_cli/configure/provider.py", line 134, in get_config
raise InvalidConfigurationError.for_profile(None)
databricks_cli.utils.InvalidConfigurationError: You haven't configured the CLI yet! Please configure by entering `/home/rohan/Documents/dk_datasci_model_pipeline/scripts/check_mlflow_connection.py configure`
Other info / logs
REPLACE_ME
What component(s) does this bug affect?
[ ] area/artifacts: Artifact stores and artifact logging
[ ] area/build: Build and test infrastructure for MLflow
Issues Policy acknowledgement
Where did you encounter this bug?
Local machine
Willingness to contribute
Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
MLflow version
poetry show mlflow-skinny
output:System information
Ubuntu 22.04.3 LTS on Windows 10 x86_64
Describe the problem
Using an mlflow client with a tracking URI pointing to a named profile and the registry UI any call that connects to the workspace fails with an authentication error.
I have the env variable DATABRICKS_CONFIG_FILE set, but not DATABRICKS_HOST or DATABRICKS_TOKEN
this gives the following output
My config file has a section for the given workspace, and the credentials are valid. If I adjust the authentication to set DATABRICKS_HOST and DATABRICKS_TOKEN instead, everything works fine
Tracking information
Code to reproduce issue
Stack trace
Other info / logs
What component(s) does this bug affect?
area/artifacts
: Artifact stores and artifact loggingarea/build
: Build and test infrastructure for MLflowarea/deployments
: MLflow Deployments client APIs, server, and third-party Deployments integrationsarea/docs
: MLflow documentation pagesarea/examples
: Example codearea/model-registry
: Model Registry service, APIs, and the fluent client calls for Model Registryarea/models
: MLmodel format, model serialization/deserialization, flavorsarea/recipes
: Recipes, Recipe APIs, Recipe configs, Recipe Templatesarea/projects
: MLproject format, project running backendsarea/scoring
: MLflow Model server, model deployment tools, Spark UDFsarea/server-infra
: MLflow Tracking server backendarea/tracking
: Tracking Service, tracking client APIs, autologgingWhat interface(s) does this bug affect?
area/uiux
: Front-end, user experience, plotting, JavaScript, JavaScript dev serverarea/docker
: Docker use across MLflow's components, such as MLflow Projects and MLflow Modelsarea/sqlalchemy
: Use of SQLAlchemy in the Tracking Service or Model Registryarea/windows
: Windows supportWhat language(s) does this bug affect?
language/r
: R APIs and clientslanguage/java
: Java APIs and clientslanguage/new
: Proposals for new client languagesWhat integration(s) does this bug affect?
integrations/azure
: Azure and Azure ML integrationsintegrations/sagemaker
: SageMaker integrationsintegrations/databricks
: Databricks integrations