allegroai / clearml-agent

ClearML Agent - ML-Ops made easy. ML-Ops scheduler & orchestration solution
https://clear.ml/docs/
Apache License 2.0
241 stars 92 forks source link

Yolo Execution with GPU #175

Open NBarile opened 1 year ago

NBarile commented 1 year ago

Hello there !

I come to you because I try to run and enqueue different yolo training remotly with clearml-agent, the task run correctly BUUUUUT, yolo doesn't run in GPU mode ...

So, have you any idea how to install CUDA (if it's the problem) in the environment created by my clearml-agent ?

I launch my agent with this code :

from ultralytics import YOLO
from clearml import Task
from clearml import Dataset

task = Task.init(
    project_name='Boudineuse',    # project name of at least 3 characters
    task_name='B1_V8', # task name of at least 3 characters
    task_type=None,
    tags=None,
    reuse_last_task_id=True,
    continue_last_task=False,
    output_uri=None,
    auto_connect_arg_parser=True,
    auto_connect_frameworks=True,
    auto_resource_monitoring=True,
    auto_connect_streams=True,    
)

task.execute_remotely(
    queue_name='danl',  # type: Optional[str]
    clone=False,  # type: bool
    exit_process=True  # type: bool
)

dataset = Dataset.get(
        dataset_id=None,  
        dataset_project="Boudineuse",
        dataset_name="Boudineuse",
        dataset_version="1.0.0",
        only_completed=True, 
        only_published=False, 
)

# Load a model
model = YOLO('yolov8n-cls.pt')  # load a pretrained model (recommended for training)

# Train the model
results = model.train(data=dataset.get_local_copy(), epochs=1000, imgsz=640)

Well, if you need more informations, please ask me.

Thanks !

jkhenning commented 1 year ago

Hi @NBarile , I see how you launch your code, but how do you run your clearml-agent?

NBarile commented 1 year ago

Oh I run it in Command prompt (cmd)

Like : clearml-agent daemon --gpu 0 --queue myqueue

jkhenning commented 1 year ago

Hi @NBarile, I see you're running it not in docker mode. The agent cannot install CUDA, and you must have CUDA installedin your system if you'd like ti to be used. The easier way to do that is to use docker mode and use a docker container with a preinstalled CUDA version.

NBarile commented 1 year ago

Ok, I gonna try with Docker. Thanks @jkhenning !

But I notice my system have CUDA installed.