[BUG] mlflow-artifacts path prefix cause error in docker

[Chesh1re]

Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

MLflow version

1.28.0

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): docker: FROM condaforge/miniforge3
• Python version: 3.8.2
• yarn version, if running the dev UI:

Describe the problem

Logging metrics and parameters works in the container, but the artifacts can not be logged.

Code works fine outside container with artifact path: "./mlruns/0/3277fd0623aa487ab732f2465b921aea/artifacts". However, inside the container, the artifact path weirdly changed to "mlflow-artifacts:/0/3277fd0623aa487ab732f2465b921aea/artifacts" automatically.

And I get the following error: mlflow.exceptions.MlflowException: The configured tracking uri scheme: 'sqlite' is invalid for use with the proxy mlflow-artifact scheme. The allowed tracking schemes are: {'https', 'http'} FileNotFoundError: [Errno 2] No such file or directory: 'mlflow-artifacts:/0/3277fd0623aa487ab732f2465b921aea/artifacts'

Tracking information

No response

Code to reproduce issue

Tracking server setup: mlflow server --host 0.0.0.0 --port 5000 --backend-store-uri sqlite:////app/MLconfigs/mlruns.db --artifacts-destination ./mlruns --serve-artifacts

Logging code: def log_artifact(self, artifact_path): """ Log artifact for each run. """ self.logger.info('Logging artifact now.') self.utils.load_tracking_uri() return mlflow.log_artifact(artifact_path)

Error happens at: Traceback (most recent call last): File "/app/src/kaiops_run.py", line 172, in ml_configs.run(runner.run_trials) File "/app/src/mlops/config_parser.py", line 63, in run _func(*_args) File "/app/src/kaiops_run.py", line 95, in run_trials dir_items = os.listdir(best_run.info.artifact_uri) FileNotFoundError: [Errno 2] No such file or directory: 'mlflow-artifacts:/0/0a808df7ac4f42ecbdd17d0542dd282c/artifacts'

Stack trace

None

Other info / logs

No response

What component(s) does this bug affect?

• [X] area/artifacts: Artifact stores and artifact logging
• [ ] area/build: Build and test infrastructure for MLflow
• [ ] area/docs: MLflow documentation pages
• [ ] area/examples: Example code
• [ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• [ ] area/models: MLmodel format, model serialization/deserialization, flavors
• [ ] area/pipelines: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
• [ ] area/projects: MLproject format, project running backends
• [ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• [ ] area/server-infra: MLflow Tracking server backend
• [ ] area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

• [ ] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• [X] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• [ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• [ ] area/windows: Windows support

What language(s) does this bug affect?

• [ ] language/r: R APIs and clients
• [ ] language/java: Java APIs and clients
• [ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

• [ ] integrations/azure: Azure and Azure ML integrations
• [ ] integrations/sagemaker: SageMaker integrations
• [ ] integrations/databricks: Databricks integrations

[harupy]

@Chesh1re Please provide the code to reproduce the issue.

[Chesh1re]

@Chesh1re Please provide the code to reproduce the issue. Tracking server setup: mlflow server --host 0.0.0.0 --port 5000 --backend-store-uri sqlite:////app/MLconfigs/mlruns.db --artifacts-destination ./mlruns --serve-artifacts

Logging code: def log_artifact(self, artifact_path): """ Log artifact for each run. """ self.logger.info('Logging artifact now.') self.utils.load_tracking_uri() return mlflow.log_artifact(artifact_path)

Error happens at: Traceback (most recent call last): File "/app/src/kaiops_run.py", line 172, in ml_configs.run(runner.run_trials) File "/app/src/mlops/config_parser.py", line 63, in run _func(*_args) File "/app/src/kaiops_run.py", line 95, in run_trials dir_items = os.listdir(best_run.info.artifact_uri) FileNotFoundError: [Errno 2] No such file or directory: 'mlflow-artifacts:/0/0a808df7ac4f42ecbdd17d0542dd282c/artifacts'

And the mlruns directory is empty. Maybe all i need to know is which API is related to "mlflow-artifacts:" this term.

[harupy]

mlflow-artifacts:/0/0a808df7ac4f42ecbdd17d0542dd282c/artifacts is not a file path. dir_items = os.listdir(best_run.info.artifact_uri) needs to be fixed. mlflow.artifacts.download_artifacts might help.

[Chesh1re]

Thank you for the supplement. Yet, I still do not know how to address this problem. Maybe I put it wrong. The problem is that "best_run.info.artifact_uri" return a "./mlflow/0/" path outside the container, yet return a "mlflow-artifacts:/0/" inside the container. Also, the "mlflow.log_artifact(local_file_path)" is not working, therefore my "./mlruns/" directory is empty.

And I don't know what cause the following MlflowException too. mlflow.exceptions.MlflowException: The configured tracking uri scheme: 'sqlite' is invalid for use with the proxy mlflow-artifact scheme. The allowed tracking schemes are: {'http', 'https'}

[harupy]

@Chesh1re Can you share the full code?

[harupy]

btw if you just want to log artifacts in mlruns, you can just run mlflow server --host 0.0.0.0 --port 5000 --backend-store-uri sqlite:////app/MLconfigs/mlruns.db.

[Chesh1re]

btw if you just want to log artifacts in mlruns, you can just run mlflow server --host 0.0.0.0 --port 5000 --backend-store-uri sqlite:////app/MLconfigs/mlruns.db.

Thank you. I address this issue by adding a no cache tag after the docker build command: " docker build -t test -f Dockerfile . --no-cache".