Can't add non-AWS s3 bucket as remote storage

[ernygati]

Describe the bug

I want to make dataset of my project to be traced by ClearMl and to be stored on non amazon AWS s3 bucket. Simply repeating the steps in the docs, I do the following:

• put access_key and secret_key in ~/clearml.conf:

     aws {
        s3 {
            # S3 credentials, used for read/write access by various SDK elements
  
            # The following settings will be used for any bucket not specified below in the "credentials" section
            # ---------------------------------------------------------------------------------------------------
            region: ""
            # Specify explicit keys
            key: "mykey"
            secret: "mysecret"
            use_credentials_chain: false
            # Additional ExtraArgs passed to boto3 when uploading files. Can also be set per-bucket under "credentials".
            extra_args: { }
            # ---------------------------------------------------------------------------------------------------
  
            credentials: [
            ]
        }
        boto3 {
            pool_connections: 512
            max_multipart_concurrency: 16
        }
    }

  • in Python import clearml Dataset and try to create a new dataset:

        `from clearml import Dataset`
        `cdataset = Dataset.create(
            dataset_name='synthetica_train',
            dataset_project='Seismic', 
            dataset_version="1.0",
            description='Synthetic data for training',
            output_uri = "s3://backet"
        )`

    But, no matter what I trying, i still get the same error: 2023-04-07 16:38:34,629 - clearml.storage - ERROR - Failed uploading: Could not connect to the endpoint URL: "https://clearml-data-test.s3.ru-central1.amazonaws.com/.clearml.test" I really can not figure out why there is always .amazonaws.com/.clearml.test added to my s3 bucket address. I tried to add endpoint parameter to clearml.config, tried to add addressing_style = path and use_fips_endpoint = my_endpointurl to ~/.aws/config file (because I found out that clearml parse this file to get deafult region of s3 storage). But I always get this ".amazonaws.com/.clearml.test"_ ending, which disable the address.

What I want is to find the way to customize the address, so that it no more amazone aws specific, and looks like https://<bucket>.my_endpoint/<key>?<parameters>? without amazone.aws. Is there a way to achieve it?

Environment

• Python 3.9
• OS Linux

  Related Discussion

  If this continues a slack thread, please provide a link to the original slack thread.

[jkhenning]

Hi @ernygati,

For clarity, this is what was answered in the Slack discussion: If this is a non-s3 the output_uri should look something like: output_uri = "s3://server-ip-here:port/backet" Then make sure you have the correct credentials in your clearml.conf file: https://github.com/allegroai/clearml/blob/a794d0d8abd0763aedce9b4ffb8c787404612f6c/docs/clearml.conf#L113

[klekass]

Hi, I noticed, that adding the output_uri like this, in the ClearML UI the output is also rendered with the IP Address of the S3 server like in the following screenshot:

This becomes problematic, once the IP Address of the Storage server changes because the path becomes invalid. Is there a way to disble rendering the server ip in the UI, when using a self-managed s3 storage server?

[jkhenning]

Hi @klekass, ClearML stores the server address (ip or other) as part of the URL since it stores a complete URL. If the address changes, that's indeed an issue - I suggest using a fixed domain name that will mask any IP change