search

allegroai / clearml-agent

ClearML Agent - ML-Ops made easy. ML-Ops scheduler & orchestration solution
https://clear.ml/docs/
Apache License 2.0
244 stars 92 forks source link

Feature: automatically install repo as pip package #195

Open markbarna opened 8 months ago

markbarna commented 8 months ago

In addition to manually listing python dependencies or using a requirements file, it would be good to have the option to install the repo as a package, if it's structured appropriately. This also would address any potential issues with inconsistent behavior of imports based on the cwd.

The specific request here is that if the repo contains a setup.cfg, setup.py, or pyproject.toml, after cloning the repo and applying any uncommitted changes, the agent will cd into the repo and run pip install ..

I was able to replicate this behavior by having the following in my code:

task = Task.init(....)
task.set_packages(["my-cool-project @ file:///root/.clearml/venvs-builds/task_repository/my-cool-project.git"])

But this seems a bit hacky, especially since I have to hard code the path to where the agent clones my repo.

jkhenning commented 8 months ago

Hi @markbarna,

Notice the root of the repo is always added to the PAYTHONPATH so it almost the equivalent of adding the repo. You can also add the repo using Task.add_requirements("-e .")

markbarna commented 8 months ago

Hi @jkhenning , I could have sworn I had already tried task.set_packages(["-e ."]), however I just tried again and that worked so I guess this does address the need. Can you explain the difference between task.add_requirements(), task.set_packages() and Task. force_requirements_env_freeze()? They all have overlapping functionality so it's not clear when I should use which one. Thank you.