Local imports when running pipelines

[abfshaal]

Describe the bug

I am trying to create a clearml pipeline from functions. I have local imports in the repo that I need to import, as in functions from other files and so on. For some reason, when I try to run the pipeline, it struggles to import these modules, so i went to check the sys.path and found that it is pointing to a tmp directory

['/private/var/folders/xm/27jjjrp13y9bq3657smh4c780000gp/T', '/opt/homebrew/anaconda3/envs/train_seg/lib/python310.zip', '/opt/homebrew/anaconda3/envs/train_seg/lib/python3.10', '/opt/homebrew/anaconda3/envs/train_seg/lib/python3.10/lib-dynload', '/opt/homebrew/anaconda3/envs/train_seg/lib/python3.10/site-packages']

However it is not pointing to the directory where I initiated the the pipeline from, is this designed by choice? and if so, how can i overcome this to create functions while still importing functions that exist locally in the repo?

To explain more, say we have this file tree |-helper_function1.py |-function1.py |-pipeline.py

in let's say I want to import the files from function1 such as

from function1 import function

this works fine, however, if function1 has the import of helper_function1.py the pipeline will not work, as the sys path wont have that during the run.

Contents of function1.py would look like

def main():
from helper_function import helper_function

this would raise the error of can't find helper_function

[AlexandruBurlacu]

Hey @abfshaal, you can specify that a specific pipeline step uses a specific repo. This way it will git clone the repository on the worker machine and all the files will be available for imports. See here.

If this doesn't work, print(os.listdir(os.getcwd())) before importing your own module/functions and attach the results here.

[abfshaal]

hi @AlexandruBurlacu , Is the repo definition supposed to work when testing locally? with the pipeline.start_locally(), or is this only supposed to take effect when executing with an agent?

[alex-burlacu-clear-ml]

AFAIK the latter option