search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.69k stars 655 forks source link

Wrong hostname when executed remotely #348

Open mctigger opened 3 years ago

mctigger commented 3 years ago

When I run clearml/examples/advanced$ python execute_remotely_example.py with 1.0 the hostname will first be my local computers name, but when the remote execution starts it will not change to the clearml-agents name.

jkhenning commented 3 years ago

Hi @mctigger,

Yeah, this is by design. The UI only shows the first reported hostname. We're planning to change that soon 🙂

mctigger commented 3 years ago

This seemed to be different before 1.0. I do not remember seeing my local machine name for scripts executed on an agent, even when started from my local machine.

jkhenning commented 3 years ago

Hmm, strange - there was no chance in the way these are reported or displayed - how do your past experiments look now when using the new server? Do you see the local machine name in them instead of the agent's name?

jkhenning commented 3 years ago

@mctigger this is a by-product of showing the host from the first line and not the last line in the report. The first line was reported from the local machine, then it started reporting from the remote machine. The change is that with ClearML SDK v1.0.x we do not reset the task as we did before (which used to clear all reports), as the server now allows enqueuing aborted tasks.

jkhenning commented 3 years ago

BTW, @mctigger, we'll change that soon so that the last line from the report is used, hence returning to the old behavior.

mctigger commented 3 years ago

Thanks a lot!