Bug: (AccessDenied) when calling the ListObjectsV2 operation

tanlin2013 commented 2 years ago

Describe the problem

When calling mlflow client to list the artifacts in runs, an error arises and says Access Denied.

Code to reproduce the issue

from mlflow.tracking.client import MlflowClient

client = MlflowClient(tracking_uri="my_uri")

run = client.search_runs(
    experiment_ids="the_id",
    filter_string="some_condition_string",
    run_view_type=ViewType.ACTIVE_ONLY,
)

artifacts = client.list_artifacts(run.info.run_id)

Error Log

An error occurred (AccessDenied) when calling the ListObjectsV2 operation.

What Happened?

Turns out MLFLOW_S3_ENDPOINT_URL pointing to the minio server needs to be set both on server and client side. MLflow doesn't do that part for you.

Reference

https://github.com/mlflow/mlflow/issues/1813

tanlin2013 commented 2 years ago

Resolved in https://github.com/tanlin2013/mbl/commit/1abe1a22657ab64117d43ad76baa62cb7d9f6a41 by puting MLFLOW_S3_ENDPOINT_URL behind the nginx server.

tanlin2013 commented 2 years ago

Encounter another error saying that AWS credentials (for minio) are wrong.

Error Logs

botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the ListObjectsV2 operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.

tanlin2013 commented 2 years ago

The variable $host and $http_host are 2 different stuffs in nginx.

tanlin2013 / mbl