How to specify a custom path to pyproject.toml and poetry.lock when running clearml-agent build?

[konstantinator]

Hi!

I use Poetry as the package manager in my project. However, I store the pyproject.toml and poetry.lock files in a separate folder named poetry (/my_hello_repo/service/poetry). When I try to restore the environment from my previous setup using the following command:

clearml-agent build --id <TASK_ID> --docker --target my_new_hello_image

I receive the following warning in the output:

Could not find pyproject.toml or poetry.lock file in /root/.clearml/venvs-builds/3.10/task_repository/my_hello_repo.git

And my code fails with an error because pip cannot build my environment.

How can I specify the path to these files in the clearml-agent build command or in the agent configuration?

UPD:

I use clearml-agent==1.8.1 and poetry==1.8.2

I've set clearml.conf

agent.package_manager.type: poetry
agent.package_manager.poetry_install_extra_args: ['--directory', '/root/.clearml/venvsbuilds/3.10/task_repository/my_hello_repo.git/service/poetry']

And I receive the following warning in the output again:

Could not find pyproject.toml or poetry.lock file in /root/.clearml/venvs-builds/3.10/task_repository/my_hello_repo.git

[TCherici]

Perhaps a bit late, but have you tried setting agent.package_manager.poetry_files_from_repo_working_dir = true in your agent configuration? from the clearml.conf:

   # if set to true, the agent will look for the "poetry.lock" file 
   # in the passed current working directory instead of the repository's root directory.
   poetry_files_from_repo_working_dir: true

It should work if you run your command from the service/poetry folder

[konstantinator]

Hi! @TCherici , thanks for your help, but it didn't work for me I receive following warning

Could not find pyproject.toml or poetry.lock file in /root/.clearml/venvs-builds/3.10/task_repository/my_hello_repo.git/my_hello_work_dir

instead

Could not find pyproject.toml or poetry.lock file in /root/.clearml/venvs-builds/3.10/task_repository/my_hello_repo.git

@jkhenning could you help me?

[konstantinator]

Hi @eugen-ajechiloae-clearml! Could you help me?

[eugen-ajechiloae-clearml]

Hi @konstantinator ! agent.package_manager.poetry_files_from_repo_working_dir = true should be the only way to tell the agent that the poetry file is not in the root of the repo at the moment. Can you make sure that the working directory of the task is the same as the directory containing your poetry files?

[konstantinator]

Hi! @eugen-ajechiloae-clearml !

I have the following structure of my repository:

/my_hello_repo
|--service
|     |--poetry
|     |--another_service_folder
|--my_hello_work_dir

The poetry files are located in a different folder than the working directory, and I want clearml to fetch the pyproject.toml and poetry.lock files from other folders.

I want to pass the path to the directory with the poetry files through

agent.package_manager.poetry_install_extra_args: ['--directory', '/root/.clearml/venvsbuilds/3.10/task_repository/my_hello_repo.git/service/poetry']

but for some reason, it doesn't work.

[TCherici]

Have you tried changing the current working directory when calling clearml-agent build ...?

The error seems to suggest that you're running the clearml-agent build ... command from inside my_hello_repo/my_hello_work_dir/ instead of my_hello_repo/service/poetry/.

[konstantinator]

Hi, @TCherici ! I tried, but it doesn't work.