allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.69k stars 655 forks source link

Using pipeline locally with ssh tunneling uses wrong IP #1053

Closed lambdaofgod closed 6 months ago

lambdaofgod commented 1 year ago

Describe the bug

I have ClearML server (with IP1) and I want to use it by tunneling from another machine M.

I did this by creating a tunnel where I redirected ClearML API ports to localhost on M - I edited clearml.conf by replacing IP1 with localhost.

When I run pipeline I get

ValueError: Could not retrieve a local copy of artifact <artifact>, failed downloading http://IP1:8081/<file_path>

Which understandably fails because there is no acces to IP1 from M.

This is confusing because when I try to get artifacts outside of pipeline (using Task) the url properly uses localhost instead of IP1.

Expected behaviour

For artifacts ClearML pipeline should use URL with IP specified in clearml.conf.

Environment

jkhenning commented 1 year ago

Hi @lambdaofgod,

I edited clearml.conf by replacing IP1 with localhost.

How would ClearML know in this case what is the correct host address?