S3 Connection setup with minIO: Cannot find credentials

[tevintchuinkam]

I am trying to setup and use minIO s3 storage like so.

task = Task.init(project_name="my_project",
                     task_name="linear-regression-test",
                     tags="testing",
                     output_uri="s3://minio.api.example.io/clearml-artifacts"
                     )

and then upload and artifact like so:

    task.upload_artifact(name=model_file,
                         artifact_object=model_file,
                         delete_after_upload=True,
                         auto_pickle=True,
                         metadata=metadata)

I followed the steps in the documentation for setting up S3 Storage with minIO and added my credentials in my ~/clearml.conf like so:

aws {
        s3 {
            key: "<minio-key>"
            secret: "<minio-secret>"
            credentials: [
                {
                    host: "minio.api.example.io:9000"
                    bucket: "clearml-artifacts"
                    key: "<minio-key>"
                    secret: "<minio-secret>"
                    multipart: false
                    secure: true
                }
            ]
        } 
}

I've many different uris for "output_uri" similarly like in this issue, but the connection cannot be established.

This is the behavior for the various values for "output_uri" that I tried:

• output_uri = minio_url: upload to the local folder with the url as name
• output_uri = minio_url/clearml-artifacts: same as above
• output_uri = https://minio_url/clearml: POST failed
• output_uri = s3://clearml: Could not get access credentials for 's3://clearml-artifacts'
• output_uri = s3://minio_url/clearml: same as above.
• output_uri = s3://minio_url:9000/clearml: same as above

I also tried setting my credentials like so, which did not help:

sdk {
    aws {
        s3 {
            credentials: [
                {
          host: "minio.api.example.io:9000"
          bucket: "clearml-artifacts"
          key: "<minio-key>"
          secret: "<minio-secret>"
          multipart: false
          secure: true
                }
            ]
        }
    }
    development.default_output_uri: "s3://minio.api.cosmic-x.idcp.inovex.io:9000/clearml-artifacts"
}

Any suggestions?

Environment

• Server type: self hosted
• ClearML SDK Version
• ClearML Server Version: WebApp: 1.10.1-359 • Server: 1.10.1-359 • API: 2.24
• Python Version 3.8.16
• OS Linux

  Related Discussion

  Uploading dataset to MinIO #391

[jkhenning]

Hi @tevintchuinkam, I think what you're looking for is:

task = Task.init(
  project_name="my_project",
  task_name="linear-regression-test",
  tags="testing",
  output_uri="s3://minio.api.example.io:9000/clearml-artifacts"
)

Assuming the minio port is 9000

[tevintchuinkam]

Hi @tevintchuinkam, I think what you're looking for is:

task = Task.init(
  project_name="my_project",
  task_name="linear-regression-test",
  tags="testing",
  output_uri="s3://minio.api.example.io:9000/clearml-artifacts"
)

Assuming the minio port is 9000

This does not fix it either sadly, I get the same error:

ValueError: Could not get access credentials for 's3://minio.api.example.io:9000/clearml-artifacts' , check configuration file ~/clearml.conf

[jkhenning]

And how did you set clearml.conf now?

[tevintchuinkam]

I tried both setups like above. I get the same error on both: ValueError: Could not get access credentials for 's3://minio.api.example.io:9000/clearml-artifacts' , check configuration file ~/clearml.conf Followed by clearml.storage - ERROR - Failed creating storage object s3://minio.api.example.io:9000/clearml-artifacts Reason: Missing key and secret for S3 storage access (s3://minio.api.example.io:9000/clearml-artifacts)

[jkhenning]

The second configuration you showed says host: "minio.api.example:9000" but the host in the error is minio.api.example.io?

[jkhenning]

I would expect it to be host: "minio.api.example.io:9000"

[tevintchuinkam]

I would expect it to be host: "minio.api.example.io:9000"

Sorry what was just a typo

[jkhenning]

Can you just try with:

sdk {
  aws {
    s3 {
      key: "<minio-key>"
      secret: "<minio-secret>"
    }
  }
}

[tevintchuinkam]

Hi @jkhenning, I just tried that (and still with output_uri="s3://minio.api.example.io:9000/clearml-artifacts" in python script) but I get the same "Could not get access credentials for 's3://minio.api.example.io:9000/clearml-artifacts'"

Just a quick sidenote: setting output_uri=True does successfully upload artifacts to the default fileserver. But this is not what I want.

[jkhenning]

It looks like the configuration file isn't loaded for some reason...

[tevintchuinkam]

It ended up working by removing the port number.

[jkhenning]

Removing from where?

[jkhenning]

Also 🔥

[Rusteam]

apparently sdk has to be a root level config. Can we update the docs here

[jkhenning]

Sure, thanks for the pointer!