[question] Is there a way to see where a dependecy version is required?

[mattangus]

What is your question?

I have two docker containers that I am using. One is for building the binaries for all the dependencies I need, and uploads them to the remote. Basically a conan file with a list of dependencies, which is the union of all the package dependencies of the other projects I use conan for. At the moment the downstream build docker is a lot bigger, than the one used for building the conan dependencies, so I don't really want to use the exact same docker image for both.

The problem I have is that the dependency graph is different (even with the exact same conanfile). Here is the setup

conanfile.py

``` from conan import ConanFile class Recipe(ConanFile): settings = "os", "compiler", "build_type", "arch" options = {"shared": [True, False], "fPIC": [True, False]} default_options = {"shared": False, "fPIC": True} def requirements(self): self.requires("fmt/10.2.1") self.requires("rapidjson/cci.20230929") self.requires("eigen/3.4.0") self.requires("opencv/4.8.1") self.requires("bitsery/5.2.3") self.requires("cnpy/cci.20180601") self.requires("libpng/1.6.42", override=True) self.requires("boost/1.84.0") self.requires("cxxopts/3.2.0") self.requires("spectra/1.0.1") self.requires("nanosvg/cci.20231025") self.requires("cub/1.17.2") self.requires("rpclib/2.3.0") self.requires("nlohmann_json/3.11.3") self.requires("gtest/1.14.0") def configure(self): if self.options.shared: self.options.rm_safe("fPIC") # modules from https://github.com/conan-io/conan-center-index/blob/master/recipes/opencv/4.x/conanfile.py#L35 opnecv_extra_modules = [ "aruco", "ccalib", "optflow", "xfeatures2d", "ximgproc", ] for mod in opnecv_extra_modules: setattr(self.options["opencv"], mod, True) self.options["opencv"].parallel = "openmp" if self.settings.arch == "x86_64": self.options["opencv"].with_gtk = True self.options["opencv"].with_wayland = False elif self.settings.arch == "armv8": self.options["opencv"].with_ffmpeg = False self.options["opencv"].with_v4l = True self.options["opencv"].with_wayland = False self.options["opencv"].with_gtk = False ```

And the graphs (output of conan graph info ... --format=dot | dot -Tsvg > graph.svg)

Downstream docker image:

Upstream docker image:

As you can see the dependency of opencv/4.8.1 has changed between the two jasper/4.1.0 -> jasper/4.2.0. My question is: is there a way to have conan tell me where the dependencies are coming from and why to debug this issue? E.g. a command like conan explain ... could output something like:

opencv/4.8.1 called require("jasper/4.1.0") on line 25

So I can figure out why the dependencies are different, looking at the conanfile.py of opencv there is no conditional inclusion of a different version of jasper so I want something to be able to track it down. Adding -vvv to the install command doesn't seem to help.

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[jcar87]

Hi @mattangus - thanks for taking the time to submit this question.

In the most recent release of Conan 2, we have updated the html format (conan graph info ... --format=html > graph.html) - if you open it in a web browser you can see exactly the dependency chain - including an easy way of identifying which references cause a dependency. Please let me know if this helps achieve what you are after.

on the other hand, since you are using Conan Center dependencies directly, bear in mind that recipe revisions can change over time, even for the same version - causing what you are seeing.

For example:

• conan install --require=opencv/4.8.1 today may require jasper/4.1.0, and a week from now it may require jasper/4.2.0 - it's still the same 4.8.1, just built with a different version.

What we advise in these instances is to use a lockfile - read more here:

• https://docs.conan.io/2.0/tutorial/versioning/lockfiles.html
• https://docs.conan.io/2.0/devops/using_conancenter.html#repeatability-and-reproducibility

Lockfiles allow you to capture a known set of working versions/revisions, such that the solver in the future will be constrainad to those, rather than arbitrarily pull a new version from the remote in the future. This is similar in functionality to what happens in other packaging tools that also operate on ranges:

• pip constraints files: https://pip.pypa.io/en/stable/user_guide/#constraints-files
• npm lockfiles: https://docs.npmjs.com/cli/v10/configuring-npm/package-lock-json
• cargo lock: https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html

[mattangus]

Thanks for the quick response!

I'll look into the html format thanks! I think what the lockfiles don't do for me is a way to separate the build and consume of the packages.

I would like to have CI pipeline build binaries for all the packages I need (including different architectures and OSes). This is done with the --build=missing flag. Then I want any consumer of packages to never need the --build=missing flag as that would be a waste of CI time since they dependencies will rarely change. I would need to somehow publish the lockfile from the build CI for it to be consumed by the downstream build pipelines. I may be missing something though.

This also feels unique to Conan as the only compiled language package manager you linked was cargo. But their moto (from my limited experience) is just compile everything all the time (i.e. --build=missing by default)

[memsharded]

I would like to have CI pipeline build binaries for all the packages I need (including different architectures and OSes). This is done with the --build=missing flag. Then I want any consumer of packages to never need the --build=missing flag as that would be a waste of CI time since they dependencies will rarely change. I would need to somehow publish the lockfile from the build CI for it to be consumed by the downstream build pipelines. I may be missing something though.

Yes, exactly, there are 2 ways to achieve immutability of dependencies as @jcar87 described:

• Using your own server with a copy of the packages that you use
• Use a lockfile. As you described, this lockfile should also go with the conanfile that other devs or users will use to do conan install, so they automatically use the locked versions, not the latest ones.

In typical production environments, it can be a mixture of both. Definitely using your own server and not relying directly on ConanCenter is recommended in most cases.

If not using these strategies, the recipes in ConanCenter do change, and are continuously updated. Even if they don't necessarily change their version, they can have their transitive dependencies change overtime, for example if boost/1.84 does a requires("zlib/[>1.2 <2], today can require zlib/1.3 and tomorrow zlib/1.3.1 is published and then boost automatically depends on it, but it might need to rebuild, so there is no way to protect users from this, except by using a lockfile or copying packages to your own server.

This also feels unique to Conan as the only compiled language package manager you linked was cargo. But their moto (from my limited experience) is just compile everything all the time (i.e. --build=missing by default)

Not only --build=missing, but they don't store binaries at all, so things will always build from source locally, binaries will never be downloaded for dependencies. They recently tried, but it seems they decided to give up, binary management and binary compatibility is quite complex, but it is very important for many of the Conan users that really need to do binary management both to reduce build time and costs, but also for other reasons like compliance or other productions requirements.