allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.43k stars 643 forks source link

fix: :bug: forward `secure` in StorageHelper for s3 #1205

Closed odulcy-mindee closed 4 months ago

odulcy-mindee commented 4 months ago

Related Issue \ discussion

After https://github.com/allegroai/clearml/commit/b78e9b904742073b30558ee06d417770aed0bc4d, secure is now default to True. We need to forward secure key otherwise, even if it has been set to False, it switches back to True.

Patch Description

At https://github.com/allegroai/clearml/blob/0594a5ac8504d090bb851265f4f5c02c8d862dde/clearml/storage/helper.py#L2069

secure key is correctly fetched from the config file.

When self._conf.update is called right after, secure is automatically changed to True.

Testing Instructions

Install latest version of clearml (1.14.3). Try with secure: false in clearml.conf under aws.s3.credentials. You should get a similar error:

[...]

File "/usr/local/lib/***3.8/site-packages/botocore/retryhandler.py", line 233, in __call__
    return self._check_caught_exception(
  File "/usr/local/lib/***3.8/site-packages/botocore/retryhandler.py", line 376, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/***3.8/site-packages/botocore/endpoint.py", line 249, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/lib/***3.8/site-packages/botocore/endpoint.py", line 321, in _send
    return self.http_session.send(request)
  File "/usr/local/lib/***3.8/site-packages/botocore/httpsession.py", line 466, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://X.X.X.X:9000/bucket/.clearml.bb4a79eb-bdbb-4c34-8e88-6b450ddc9def.test [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)

During handling of the above exception, another exception occurred:

[...]

  File "/usr/local/lib/***3.8/site-packages/clearml/task.py", line 609, in init
    task.output_uri = str(cls.__default_output_uri)
  File "/usr/local/lib/***3.8/site-packages/clearml/task.py", line 1207, in output_uri
    helper.check_write_permissions(value)
  File "/usr/local/lib/***3.8/site-packages/clearml/storage/helper.py", line 2789, in check_write_permissions
    raise ValueError("Insufficient permissions (delete failed) for {}".format(base_url))
ValueError: Insufficient permissions (delete failed) for s3://X.X.X.X:9000/bucket/