Import issue with run_locally

[Arnechos]

Hello, so far I really like the library. Right now I'm testing the pipeline functions and I've hit a wall.

I have a directory structure as follows:

• root
• • src
• • • definition
• • • tasks

definition dir contains python file main_pipeline.py

from clearml.automation.controller import PipelineDecorator
from src.tasks.feature_generation import feature_generation_step

@PipelineDecorator.pipeline(
    name="main_pipeline",
    project="tests",
    version="0.1",
)
def execute(arg):
    df = feature_generation_step(arg)

    print(f"Data frame shape={df.shape}")

tasks dir contain python file feature_generation.py

from clearml.automation.controller import PipelineDecorator
from clearml import TaskTypes

@PipelineDecorator.component(
    return_values=["df"],
    cache=True,
    task_type=TaskTypes.data_processing,
    # execution_queue="default",
)
def feature_generation_step(arg):
    import src.tasks.dummy as dm
    dm.foo()
    df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
    return df

main.py

from clearml.automation.controller import PipelineDecorator
from src.definition.main_pipeline import execute
if __name__ == "__main__":
    PipelineDecorator.run_locally()

    execute(
        arg=None
    )

    print("process completed")

clearml pipeline fails each time I try to run it on the line with import src.tasks.dummy. No matter how I've changed the path/added init I keep getting ModuleNotFoundError. Is there a way to import function from another file into step function?

[alex-burlacu-clear-ml]

Hey @Arnechos, glad you like ClearML. About your issue, have you tried running everything without any ClearML pipeline decorators as a sanity check to make sure it’s a ClearML issue?

In general, when you want to reference your own files inside a pipeline step you either need (1) to have your whole project in a git repository, and then ClearML will be able to infer how to access the module, or you need to (2) make all the stuff in src a package published on PyPI and specify it in requirements.

I would highly advise using the 1st approach. This is done because we want to keep our local and remote execution similar.

[Arnechos]

@alex-burlacu-clear-ml

Hello, code itself does work, also when running PipelineDecorator.debug_pipeline(). I did tests with printing os.getcwd() inside feature_generation_step and the path was correct.

1. The project is in the git repo, root is a directory separated from the rest of the model training code.

[yur1xpp]

@alex-burlacu-clear-ml I have my whole project in git repo, but it still failed to install them properly, any advice? Here's the log, notice the line ModuleNotFoundError: No module named 'mypackage'

Python executable with version '3.11' requested by the Task, not found in path, using '/usr/bin/python3' (v3.10.12) instead
::: Using Cached environment /home/user/.clearml/venvs-cache/3af663243254355f2ed925b50841839b5af6b.40b88e8a71a459462d0cf971c414214a :::
Using cached repository in "/home/user/.clearml/vcs-cache/mypackage.d0127e09464564564565697a05f4131/mypackage"
Note: switching to 'df7edca5d29xxxxxxxxxx16ec63918eb'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
  git switch -c <new-branch-name>
Or undo this operation with:
  git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at df7edca tx4
type: git
url: ssh://git-codecommit.server.amazonaws.com/v1/repos/mypackage
branch: HEAD
commit: df7edca5d29xxxxxxxxxx16ec63918eb
root: /home/user/.clearml/venvs-builds.2/3.10/task_repository/mypackage
Adding venv into cache: /home/user/.clearml/venvs-builds.2/3.10
Running task id [a1513d2fc76c4581be34324326d2f79]:
[.]$ /home/user/.clearml/venvs-builds.2/3.10/bin/python -u myscript.py
Summary - installed python packages:
pip:
.
.
.
- tqdm==4.66.2
- urllib3==2.2.1
Environment setup completed successfully
Starting Task Execution:
Traceback (most recent call last):
  File "/home/user/.clearml/venvs-builds.2/3.10/task_repository/mypackage/myscript.py", line 3, in <module>
    from mypackage.constants import fun1, func2
ModuleNotFoundError: No module named 'mypackage'
Leaving process id 39608
DONE: Running task 'a1513d2drftgrete155c136d2f79', exit status 1
Process failed, exit code 1

[eugen-ajechiloae-clearml]

Hi @yur1xpp! Can your repo be installed as a pip package? If so, you could set the packages to contain a link to that repository using https://clear.ml/docs/latest/docs/references/sdk/task#set_packages. For example: task.set_packages(["git+https://github.com/allegroai/clearml", "other_reqs_here"]) You could also add this to your requirements.txt and use the requirements file instead

[yur1xpp]

@eugen-ajechiloae-clearml they are hosted on private aws code commit. At the moment, clearml will capture my environment (which indicates that my project is git hosted), and when run on remotely on agent, it'll pull from the code commit repo (ssh://). This however makes it problematic, because it can't do the from mypackage import ... line as it's not installed as editable, and also failed when do it with relative import (it'll report this line for example from .. import func1 that is in __init__.py as from None import func1.