[BUG] Unable to load Articrafts in UI - AJAX Internal server error 500

SaiKumarNerella commented 2 years ago

Thank you for submitting an issue. Please refer to our issue policy for additional information about bug reports. For help with debugging your code, please refer to Stack Overflow.

Please fill in this bug report template to ensure a timely and thorough response.

Willingness to contribute

The MLflow Community encourages bug-fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

[ ] Yes. I can contribute a fix for this bug independently.
[X ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
[ ] No. I cannot contribute a bug fix at this time.

System information

Have I written custom code (as opposed to using a stock example script provided in MLflow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux
MLflow installed from (source or binary): Pip
MLflow version (run mlflow --version): 1.24.0
Python version: 3.8.10
npm version, if running the dev UI:
Exact command to reproduce: mlflow server --backend-store-uri mysql+pymysql://${MYSQL_USER}:${MYSQL_PASSWORD}@${MYSQL_HOST}:3306/${MYSQL_DATABASE} --default-artifact-root https://objectstorage.ca-toronto-1.oraclecloud.com/${OCI_BUCKET_URI}/ -h 0.0.0.0 ( object storage on orcale cloud and data storage on MySQL)

Describe the problem

Describe the problem clearly here. Include descriptions of the expected behavior and the actual behavior.

Articrafts Tab on the Run Details page failed to load the articrafts from the oracle cloud object storage.

Code to reproduce the issue

Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

2022/04/01 17:19:04 ERROR mlflow.server: Exception on /ajax-api/2.0/preview/mlflow/artifacts/list [GET]
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/mlflow/server/handlers.py", line 327, in wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/mlflow/server/handlers.py", line 385, in wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/mlflow/server/handlers.py", line 676, in _list_artifacts
    artifact_entities = _get_artifact_repo(run).list_artifacts(path)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/mlflow/store/artifact/http_artifact_repo.py", line 44, in list_artifacts
    url, tail = self.artifact_uri.split(endpoint, maxsplit=1)
ValueError: not enough values to unpack (expected 2, got 1)

What component(s), interfaces, languages, and integrations does this bug affect?

Components

[X] area/artifacts: Artifact stores and artifact logging
[ ] area/build: Build and test infrastructure for MLflow
[ ] area/docs: MLflow documentation pages
[ ] area/examples: Example code
[ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
[ ] area/models: MLmodel format, model serialization/deserialization, flavors
[ ] area/projects: MLproject format, project running backends
[ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
[ ] area/server-infra: MLflow Tracking server backend
[ ] area/tracking: Tracking Service, tracking client APIs, autologging

Interface

[ X] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
[ ] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
[ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
[ ] area/windows: Windows support

Language

[ ] language/r: R APIs and clients
[ ] language/java: Java APIs and clients
[ ] language/new: Proposals for new client languages

Integrations

[ ] integrations/azure: Azure and Azure ML integrations
[ ] integrations/sagemaker: SageMaker integrations
[ ] integrations/databricks: Databricks integrations

BenWilson2 commented 2 years ago

Do the artifacts appear on OCI object store (do they actually get written?)

SaiKumarNerella commented 2 years ago

Yes

DougTrajano commented 2 years ago

I'm facing the same issue with version 1.26.1

dbczumar commented 2 years ago

@BenWilson2 @harupy Any updates here?

I've added the help wanted label, as it would be very helpful to get community assistance root causing and fixing this issue. Thank you!

DougTrajano commented 2 years ago

Hi folks!

In my case, the host (App Runner) was using a VPC that doesn't have a VPC Endpoint configured to Amazon S3. I added it and the issue was fixed.

Rusteam commented 1 year ago

we had a similar issue on mlflow==1.29.0 but it was because of wrong artifact location. S3 path was erroneous: ss3://path/to/object. After fixing it to s3://path/to/object all works fine.

mlflow / mlflow