allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.43k stars 643 forks source link

Failed to execute local process by clearml-task #1216

Closed edenbuaa closed 4 months ago

edenbuaa commented 4 months ago

The task was always in pending when I executed my local task by clearml-task.

Steps to reproduce:

  1. Deploy clearml in self-hosted mode, and everything is working.
  2. Run a local container and execute the local script by clearml-task clearml-task --script task.py --project test --name test --requirements requirements.txt --queue default
  3. Check the task in the webapp, and this task is pending.

Is this the correct command to execute the local script in its local environment with clearml-task? if not, how to do for this case.

Furthermore, Task.init work well, but I do not want to edit the task.py.

ainoam commented 4 months ago

@edenbuaa Your description seems to indicate there's no ClearML agent servicing the default queue.

Does this help?

edenbuaa commented 4 months ago

@ainoam yes, there's no agent. I think it should work even without an agent.

In fact, I only want to run the task.py in my container, and clearm server can record and log this task by clearml-task. (if I use the Task.init, it works)

ainoam commented 4 months ago

@edenbuaa clearml-task is for creating an entry on the server to be subsequently used e.g. execute on a remote machine. When you invoke it like you did, you enqueue it on the default queue, hence you need a ClearML agent that will pull and execute your task.

For executing a script locally, clearml-task is not required and Task.init() is the way to go.

edenbuaa commented 4 months ago

@ainoam Got it. thanks.

Is there a way to execute a script locally without modifying the codes?

ainoam commented 4 months ago

@edenbuaa Unfortunately no - you either instrument your code yourself for local execution, or ClearML will do it for you through clearml-task when executing remotely.

edenbuaa commented 4 months ago

ah, Okay, but I still hope that this feature can be added in the later. @ainoam