gnutls_handshake() failed: An unexpected TLS

[Arseny1998]

Hi. I'm using self-hosted clearml-server and trying to setup clearml-agent on my machine. After I enuque a task if failed with fatal: unable to access 'https://gitlab....::27022:...../toy_mlops_project.git/' gnutls_handshake() failed: An unexpected TLS packet was received. I added git key for private repository to credentials. I think it might be related to 27022 port, but I don't know to delete from url

[jkhenning]

Hi @Arseny1998,

Can you please include the complete log?

[Arseny1998]

cloning: ssh://gitlab...._project.git Using user/pass credentials - replacing ssh url 'ssh://gitlab.....project.git' with https url 'https://gitlab.....project.git' fatal: unable to access 'https://gitlab.....project.git/': gnutls_handshake() failed: An unexpected TLS packet was received. Repository cloning failed: Command '['clone', 'https://gitlab....project.git', '/home/arseny/.clearml/vcs-cache/project.git.472150a4/project.git', '--recursive', '--quiet']' returned non-zero exit status 128. clearml_agent: ERROR: Failed cloning repository. 1) Make sure you pushed the requested commit: (repository='ssh://gitlab....project.git', branch='main', commit_id='3c34e3caa......d4e', tag='', docker_cmd=None, entry_point='train_xgboost_with_hyp.py', working_dir='hyperparam_optimization') 2) Check if remote-worker has valid credentials [see worker configuration file]

[jkhenning]

I need to see the complete log that includes the agent version and other settings

[Arseny1998]

sdk.log.null_log_propagate = false sdk.log.task_log_buffer_capacity = 66 sdk.log.disable_urllib3_info = true sdk.development.task_reuse_time_window_in_hours = 72.0 sdk.development.vcs_repo_detect_async = true sdk.development.store_uncommitted_code_diff = true sdk.development.support_stopping = true sdk.development.default_output_uri = sdk.development.force_analyze_entire_repo = false sdk.development.suppress_update_message = false sdk.development.detect_with_pip_freeze = false sdk.development.worker.report_period_sec = 2 sdk.development.worker.ping_period_sec = 30 sdk.development.worker.log_stdout = true sdk.development.worker.report_global_mem_used = false agent.worker_id = arseny-Dell-G15-5510:0 agent.worker_name = arseny-Dell-G15-5510 agent.force_git_ssh_protocol = false agent.python_binary = agent.package_manager.type = pip agent.package_manager.pip_version.0 = <20.2 ; python_version < '3.10' agent.package_manager.pip_version.1 = <22.3 ; python_version >\= '3.10' agent.package_manager.system_site_packages = false agent.package_manager.force_upgrade = false agent.package_manager.conda_channels.0 = pytorch agent.package_manager.conda_channels.1 = conda-forge agent.package_manager.conda_channels.2 = defaults agent.package_manager.priority_optional_packages.0 = pygobject agent.package_manager.torch_nightly = false agent.package_manager.poetry_files_from_repo_working_dir = false agent.venvs_dir = /home/arseny/.clearml/venvs-builds agent.venvs_cache.max_entries = 10 agent.venvs_cache.free_space_threshold_gb = 2.0 agent.venvs_cache.path = ~/.clearml/venvs-cache agent.vcs_cache.enabled = true agent.vcs_cache.path = /home/arseny/.clearml/vcs-cache agent.venv_update.enabled = false agent.pip_download_cache.enabled = true agent.pip_download_cache.path = /home/arseny/.clearml/pip-download-cache agent.translate_ssh = true agent.reload_config = false agent.docker_pip_cache = /home/arseny/.clearml/pip-cache agent.docker_apt_cache = /home/arseny/.clearml/apt-cache agent.docker_force_pull = false agent.default_docker.image = nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04 agent.enable_task_env = false agent.hide_docker_command_env_vars.enabled = true agent.hide_docker_command_env_vars.parse_embedded_urls = true agent.abort_callback_max_timeout = 1800 agent.docker_internal_mounts.sdk_cache = /clearml_agent_cache agent.docker_internal_mounts.apt_cache = /var/cache/apt/archives agent.docker_internal_mounts.ssh_folder = ~/.ssh agent.docker_internal_mounts.ssh_ro_folder = /.ssh agent.docker_internal_mounts.pip_cache = /root/.cache/pip agent.docker_internal_mounts.poetry_cache = /root/.cache/pypoetry agent.docker_internal_mounts.vcs_cache = /root/.clearml/vcs-cache agent.docker_internal_mounts.venv_build = ~/.clearml/venvs-builds agent.docker_internal_mounts.pip_download = /root/.clearml/pip-download-cache agent.apply_environment = true agent.apply_files = true agent.custom_build_script = agent.disable_task_docker_override = false agent.git_user = Arseny agent.default_python = 3.10 agent.cuda_version = 0 agent.cudnn_version = 0 api.version = 1.5 api.verify_certificate = true api.default_version = 1.5 api.http.max_req_size = 15728640 api.http.retries.total = 240 api.http.retries.connect = 240 api.http.retries.read = 240 api.http.retries.redirect = 240 api.http.retries.status = 240 api.http.retries.backoff_factor = 1.0 api.http.retries.backoff_max = 120.0 api.http.wait_on_maintenance_forever = true api.http.pool_maxsize = 512 api.http.pool_connections = 512 api.api_server = http://12.12.12.12:8008 api.web_server = http://12.12.12.12:8080 api.files_server = http://12.12.12.12:8081 api.credentials.access_key = ADFSDFS134FSD api.host = http://12.12.12.12:8008 Executing task id [eda30de2a916d7108d3743]: repository = ssh://gitlab.....project.git branch = main version_num = 3c34e3caad8e7fd78s63675aaa798997c310b6d4e tag = docker_cmd = entry_point = train_xgboost_with_hyp.py working_dir = hyperparam_optimization created virtual environment CPython3.10.7.final.0-64 in 193ms creator CPython3Posix(dest=/home/arseny/.clearml/venvs-builds/3.10, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/arseny/.local/share/virtualenv) added seed packages: pip==23.3.2, setuptools==69.0.3, wheel==0.42.0 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator cloning: ssh://gitlab.....net:27022/...project.git Using user/pass credentials - replacing ssh url 'ssh://gitlab....project.git' with https url 'https://gitlab....:27022....project.git' fatal: unable to access 'https://gitlab....:27022....project.git/': gnutls_handshake() failed: An unexpected TLS packet was received. Repository cloning failed: Command '['clone', 'https://gitlab....:27022/....project.git', '/home/arseny/.clearml/vcs-cache/toy_mlops_project.git.472150a4211a84b85bb33fd33jje6bc9ca/project.git', '--recursive', '--quiet']' returned non-zero exit status 128. clearml_agent: ERROR: Failed cloning repository. 1) Make sure you pushed the requested commit: (repository='ssh://gitlab.....:27022/....project.git', branch='main', commit_id='3c34e3caasad898997c310b6d4e', tag='', docker_cmd=None, entry_point='train_xgboost_with_hyp.py', working_dir='hyperparam_optimization') 2) Check if remote-worker has valid credentials [see worker configuration file]