[question] Download Package via Python API

[GRyzer]

What is your question?

We are currently building an image with the help of a conanfile, in which we specify the packages as dependencies that we want to have in an image. Our problem is that we have to pass certain information from these packages into a poetry module (which cannot be changed). Currently we need a bridge between Conan and Poetry. So if we could write a poetry module that could use Conan and download packages from Artifactory, I think our problem would be solved. Is there a possibility to achieve the same as a Conanfile via the Python API? The only requirement would be that we can download the package that matches a build/host profile.

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @GRyzer

Yes, the PythonAPI is now available and documented in Conan 2: https://docs.conan.io/2/reference/extensions/python_api.html

The documentation and stabilization is a work in progress, but you can see full details of its usage in the implementation of the current commands.

The CommandAPI is the higher-level API that allows to call Conan commands from the API more easily: https://docs.conan.io/2/reference/extensions/python_api/CommandAPI.html, with something like conan_api.command.run().

The only requirement would be that we can download the package that matches a build/host profile.

Note that this is not exactly how it works. You can download a package by its full reference, including its package_id, or you can install packages using profiles as inputs. But downloading just a package that matches a profile is not possible, because there is no a 1:1 relationship between profiles and package_id, and to compute the mapping it is necessary to evaluate the dependency graph.

[GRyzer]

Ok is it possible to make an example. In the Conanfile.py i have following dependencies:

def requirements(self):
    self.requires("foo/[~2.1]")

How would i be able to get this package from my artifactory. Should i do something like this:

from conan.api.conan_api import ConanAPI

def download_foo():
    conan_api = ConanAPI()
    conan_api.command.run("conan download foo/[~2.1] -p os=Windows AND compiler=clang")

Honestly i am a bit lost here. I cant figure out by documentation or by the source code itself, how i can get the package i would like to have. Lets say i have created foo with this command: conan create . -pr:b=windows-build -pr:h=linux-host --version=2.1 and afterwards i upload it to my remote. How would i be able to retrieve exactly this package?

[memsharded]

That is the thing, I think it is first necessary to clarify the behavior and what is the desired functionality before being able to implement it.

foo/[~2.1] for example is not a package, it is just a requirement that can match multiple versions. When the requirement is evaluated it will be first resolved to some specific version like foo/2.1.2, by default the latest available in the range if there is no other requirement fixing other version. For that version it will also resolve to the latest recipe-revision, something like foo/2.1.2#recipe_revision1.

This first step is only the "recipe", but not yet the package binary. Every recipe can resolve to multitude of package binaries (or packages in short), for example there will be packages for Windows-msvc-193-release-x86_64 and for Linux-gcc-11-debug-amrv8. But the problem is that this is still not enough to determine the package binary (defined by its package_id). For a given configuration defined in a profile, like "Windows-msvc-193-release-x86_64", there can also be multiple different binaries. This is because different versions of the dependencies will also generate different binaries. If the package is not foo, but for example boost, it will have transitive dependencies zlib and bzip2. The same boost/1.84.0 version with exactly the same profile will produce different package_id with different binaries when using different zlib and bzip2 versions.

So in order to get a specific binary/package_id from a profile it is necessary to evaluate the dependency graph. This is done with a conan install not a conan download. The conan download uses an already resolved package_id, but not a profile.

Furthermore, when using some library like boost, if we are using it as a static library we will also need zlib and bzip2 static libraries it is not enough to download only boost. Same happens if we decide to use dependencies as shared libraries. So generally downloading a library package with conan download is not enough to use it, and we need to bring also its transitive dependencies.

Does this clarify why conan download foo/[~2.1] -p os=Windows AND compiler=clang is not possible? It is not a matter of the interface, it is related to dependency management with C++ packages.

[GRyzer]

That is the thing, I think it is first necessary to clarify the behavior and what is the desired functionality before being able to implement it.

It's actually a pretty straight-forward task. Download the latest version of foo/[~2.1] which has been created with a specific build and host profile.

Your explanation describes very well, why i am unable to use conan download. I now know why I shouldn't use conan download, but not what exactly I should use instead. A conan install? However, I thought that I always need a conanfile.py to run a conan install. Doesn't this contradict the requirement to only download packages from Artifactory via the Conan Python API? Should i do something like this:

    conan_api = ConanAPI()
    cli = Cli(conan_api)
    cli.add_commands()
    conan_api.command.cli = cli
    conan_api.command.run(["remove", "*", "-c"])
    conan_api.command.run(["install", ".", "-pr:b=windows-build", "-pr:h=linux-host"])

with this conanfile:

class BarRecipe(ConanFile):
    name = "bar"
    version = "0.0.0.3"

    def requirements(self):
        self.requires("foo/[~2.1]")

If this is the correct approach, how do i get access to the package folder of foo?

[memsharded]

I now know why I shouldn't use conan download, but not what exactly I should use instead. A conan install? However, I thought that I always need a conanfile.py to run a conan install. Doesn't this contradict the requirement to only download packages from Artifactory via the Conan Python API?

No, you don't need a conanfile for that, you can use:

conan install --requires=foo/[>1.0] -pr:h=myprofile ...

Getting access to the cache folder containing the different packages can be done in several ways:

• The output of the install method will contain a graph and it contains all the information, including the package-folder
• The conan cache path returns paths in the cache for a given reference
• It is possible to use deployers to collect files from the cache, not only for a given reference, as said above, it is possible that you need to collect the libraries from that package and its dependencies.

Check for example:

from conan.api.conan_api import ConanAPI
from conan.cli.cli import Cli

conan_api = ConanAPI()
cli = Cli(conan_api)
cli.add_commands()
conan_api.command.cli = cli
result = conan_api.command.run(["install", "--requires=zlib/[>1.2 <2]", "-pr:b=default",
                                "-pr:h=default", "--update"])
print(result["graph"].serialize())

You will see the information you want in that json

conan_api.command.run(["remove", "*", "-c"])

Probably this is inefficient and you don't want to do it. If you want to check things are updated from the server, pass the --update argument.

[GRyzer]

Thank you for clarification! I am able to achieve my goal with given informations!

conan_api.command.run(["remove", "*", "-c"])

Probably this is inefficient and you don't want to do it. If you want to check things are updated from the server, pass the --update argument.

Yeah you are right. My purpose here was to clear the cache in order to save some hard disk space :D

[memsharded]

Happy to help!