search

mlflow / mlflow

Open source platform for the machine learning lifecycle
https://mlflow.org
Apache License 2.0
18.53k stars 4.21k forks source link

Not able to log artifacts and models using MLFlow on OVH managed Object storage #8571

Open altruistcoder opened 1 year ago

altruistcoder commented 1 year ago

Hello,

We have an MLflow instance which is running inside a container on our kubernetes cluster. The backend store for the instance is being provided through a PostgreSQL pod and the artifact store for the instance is one of the S3 buckets provided by OVH managed object storage service.

The problem which we are facing currently is that MLflow is not able to log artifacts and/or models in the specified S3 bucket and giving the following error:

EndpointConnectionError: Could not connect to the endpoint URL: "http://s3.<region>.io.cloud.ovh.net/<bucket-name>/<path-in-the bucket>"

We are also setting all required environmental variables at both the client and server end like:

but still it doesn't seem to work.

We have already tried a lot of fixes that were provided on the internet but nothing seems to work for our case.

Can you please help us to understand if we are missing some information or why it's not able to connect to the bucket.

harupy commented 1 year ago

@altruistcoder Can you log artifacts without mlflow?

altruistcoder commented 1 year ago

@harupy

I tried uploading some files directly through boto3 into my S3 bucket and I am able to do so by specifying the region explicitly while creating the boto3 resource and/or client object.

mlflow-automation commented 1 year ago

@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.