search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.61k stars 651 forks source link

SSLError with file uploads #1107

Closed abfshaal closed 1 year ago

abfshaal commented 1 year ago

Describe the bug

When trying to upload files to clearml I am getting this error SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')) even tho I have the api.verify set to false

The clearml deployment I have is using the helm chart deployment, with the version of clearml 1.10.4 for server, and the source code for launching the data upload on version 1.10.3.

If I add a s3 storage, some part of the data upload works, and another starts showing the retrying error message Retrying (Retry(total=2, connect=5, read=5, redirect=5, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))'

If I export the environment variable CURL_CA_BUNDLE to an empty string, it works, but is there any other better solution than this workaround ?

valeriano-manassero commented 1 year ago

Hi, having CURL_CA_BUNDLE set to empty is just a workaround to avoid certificate validation. Best practice is to use valid certificates, if in your case they are produced internally, you need to have CURL_CA_BUNDLE set to file containing CA bundle so system is able to trust it.