search

nv-morpheus / Morpheus

Morpheus SDK
Apache License 2.0
364 stars 136 forks source link

[BUG]: databricks_cli import missing from main CLI in 24.03 release #1600

Closed pdmack closed 7 months ago

pdmack commented 7 months ago

Version

24.03

Which installation method(s) does this occur on?

Docker

Describe the bug.

After building and running the 24.03 release image, the morpheus CLI tool fails to run a pipeline due to not finding the databricks_cli module. However, the relevant packages appear to be in the env:

# conda list | grep databricks
databricks-cli            0.217.0              ha8f183a_0    conda-forge
databricks-connect        14.3.1                   pypi_0    pypi
databricks-sdk            0.24.0                   pypi_0    pypi

Minimum reproducible example

morpheus run pipeline-ae

Relevant log output

Click here to see error details

  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/stages/preprocess/deserialize_stage.py", line 28, in 
    from morpheus.modules.preprocess.deserialize import DeserializeLoaderFactory
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/modules/__init__.py", line 26, in 
    from morpheus.modules import mlflow_model_writer
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/modules/mlflow_model_writer.py", line 20, in 
    from morpheus.controllers.mlflow_model_writer_controller import MLFlowModelWriterController
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/controllers/mlflow_model_writer_controller.py", line 21, in 
    import mlflow
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/__init__.py", line 44, in 
    from mlflow import (
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/artifacts/__init__.py", line 11, in 
    from mlflow.tracking import _get_store
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/tracking/__init__.py", line 8, in 
    from mlflow.tracking._model_registry.utils import (
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/tracking/_model_registry/utils.py", line 15, in 
    from mlflow.store._unity_catalog.registry.rest_store import UcModelRegistryStore
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/store/__init__.py", line 1, in 
    from mlflow.store.artifact import artifact_repo
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/store/artifact/artifact_repo.py", line 14, in 
    from mlflow.utils.file_utils import ArtifactProgressBar, create_tmp_dir
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/utils/file_utils.py", line 51, in 
    from mlflow.utils.databricks_utils import _get_dbutils
  File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/utils/databricks_utils.py", line 7, in 
    from databricks_cli.configure import provider
ModuleNotFoundError: No module named 'databricks_cli'

Full env printout

Click here to see environment details

```
 # conda list | grep databricks
databricks-cli            0.217.0              ha8f183a_0    conda-forge
databricks-connect        14.3.1                   pypi_0    pypi
databricks-sdk            0.24.0                   pypi_0    pypi
```

Other/Misc.

No response

Code of Conduct