search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.43k stars 643 forks source link

Allow to set the `working_dir` with pipelines from decorators #1194

Closed materight closed 1 month ago

materight commented 5 months ago

Proposal Summary

Add a working_dir parameter to PipelineDecorator.pipeline and PipelineDecorator.component to be able to set the execution working_dir.

Motivation

Currently when defining a pipeline using PipelineDecorator.component, it's only possible to set the repo for the script execution, but not the working_dir. Therefore it defaults to ., and if all the code is inside a subfolder, when running the pipeline remotely the script it's not able to find any local modules.

Related Discussion

Original slack thread.

ainoam commented 5 months ago

Thanks for suggesting @materight. We'll try to address this in a near version.

pollfly commented 1 month ago

Hey @materight! v1.15.0 is now out supporting custom working directory for pipelines

materight commented 1 month ago

Great! Thanks a lot