Clearml Task: pip install order

[mathrb]

Hello

I've got a dependency issue when running a task. Requirements have to be installed in a specific order: I've got a dependency on detectron2, which must be installed from source (pip install git+https://github.com/facebookresearch/detectron2.git). I cannot install this package with the prebuilt version since the prebuilt only works torch 1.10 This package has a setup that requires torch to be already installed. The problem I have is that detectron2 setup is done before torch installation. Is there a way to achieve this? Today I create my task using:

 train_detectron2_task = Task.create(
        project_name=BASE_TEMPLATE_PROJECT,
        repo=REPO,
        branch=BRANCH,
        script="train_detectron2.py",
        requirements_file="requirements/requirements.txt",
        packages=detectron2_packages,
        add_task_init_call=False,
        task_name="train_detectron2",
        task_type="training",
        docker="local_report/custom-image:latest",
        working_directory=WORKING_DIRECTORY
    )

detectron2_packages is a list of dependencies containing detectron2. I thought that maybe packages are installed after requirements_file, but this is not the case.

Thanks for your help

[jkhenning]

Hi @mathrb ,

When using Task.create() the code uses the provided list of packages and requirements file to construct a list of required packages. When the task is executed remotely by the ClearML Agent, this list is passed to pip which installs it (and has no guarantee on the order). You have several options:

• Have this preinstalled in your docker file, and in the agent settings set agent.package_manager.system_site_packages: true
• Use the docker_bash_setup_script argument to preinstall this package(s) in the docker base python and in the agent settings set agent.package_manager.system_site_packages: true
• Use the agent's agent.package_manager.priority_packages or agent.package_manager.priority_optional_packages to make sure the agent will attempt to install these packages before installing the task's required packages

[mathrb]

Hello @jkhenning Thanks for you answer. Regarding the latest option agent.package_manager.priority_packages Does it install the package defined in this priority list, or does it tries to find in the task requirements a matching package. The reason for this being: will it install a fixed version, or will it use the package version as specified in the task requirements.

Regarding the first option with agent.package_manager.system_site_packages: true, what could be the drawbacks of using the system site packages in this clearml docker agent environment?

[jkhenning]

Regarding the first option with agent.package_manager.system_site_packages: true, what could be the drawbacks of using the system site packages in this clearml docker agent environment?

No drawbacks I think - it will basically just use a preinstalled package if it meets the criteria

[jkhenning]

Regarding the latest option agent.package_manager.priority_packages Does it install the package defined in this priority list, or does it tries to find in the task requirements a matching package. The reason for this being: will it install a fixed version, or will it use the package version as specified in the task requirements.

It should find them in your requirements, and if there (searched by name, not version) it will install them first. Order in this list matters (i.e. priority packages will be installed by the order they're specified).

[mathrb]

@jkhenning thanks for you answers. Will start with the agent.package_manager.priority_packages which looks better in my case.