[bug] Overriding requires of tool_requires does not work

[Psy-Kai]

I have package (using a conanfile.py) which has multiple tool requires. One is protobuf and another one if a custom tool which requires libarchive.
Now since this commit bumped zlib in protobuf my package does not build anymore. And I cannot find any way to override the requirements.

The documentation states that "As a rule of thumb, downstream defined values always override upstream dependency values" but even if I add zlib/1.2.12 as tool_requires in my conanfile.py I get an error.

Environment Details (include every applicable attribute)

• Operating System+version: KUbuntu 21.04
• Compiler+version: gcc 10.3
• Conan version: 1.47.0
• Python version: 3.9.5

Steps to reproduce (Include if Applicable)

Create conanfile.py with

    tool_requires = [
        "protobuf/3.19.2",
        "libarchive/3.6.0",
    ]

Logs (Executed commands with output) (Include/Attach if Applicable)

``` ERROR: Conflict in libarchive/3.4.0: 'libarchive/3.4.0' requires 'zlib/1.2.11' while 'protobuf/3.19.2' requires 'zlib/1.2.12'. To fix this conflict you need to override the package 'zlib' in your root package. ``` or (when adding `zlib` explicit to `tool_requires`) ``` ERROR: Conflict in libarchive/3.4.0: 'libarchive/3.4.0' requires 'zlib/1.2.11' while 'conanfile.py (MyPackage/1.2.3)' requires 'zlib/1.2.12'. To fix this conflict you need to override the package 'zlib' in your root package. ```

[franramirez688]

Hi @Psy-Kai

Thanks for reporting it. I guess the problem is coming because those tool requirements require zlib directly and not as another tool_requirement.

These dependencies used as a normal requirements work:

    def requirements(self):
        self.requires("zlib/1.2.12", override=True)
        self.requires("protobuf/3.19.2")
        self.requires("libarchive/3.6.0")

Actually, I don't know if it's a bug so we'll need a bit more to investigate the issue.

[Psy-Kai]

The same problem could occur when the qt moc and the protobuf protoc is needed (Both packages contain zlib as normal requires). In that case the tool_requires should/has to be used.

Additionally: since I have a cross compiled project I definitely cannot use normal requires.

[memsharded]

Yes, this is known, and by design. tool_requires dependencies are private to each package, they cannot be changed by other packages, because that would break many package builds. For overriding tool_requires, they need to be overriden in profiles directly.

Conan 2.0 behavior is still similar. As tool_requires by default have the visible=False trait, they are not visible from other downstream dependencies, and their overrides don't apply there. tool_requires can be made visible to the consumers, so they can be overriden, but that also means that all packages in that subgraph should use the same version of the tool_requires or they will conflict. But this is something that the user can decide to implement either way. What is not possible is the mixture, it is not possible to override things that are not defined as visible, that is exactly what the trait means.

[Psy-Kai]

@memsharded I think you missunderstand my problem. For an easier explanation a graph:

zlib/1.2.11       zlib/1.2.12
    ^                  ^
    |                  |
    |     requires     |
    |                  |
   qt              protobuf
    ^                  ^
    |                  |
    |  tool_requires   |
    +--------+---------+
             |
          MyPackage

Currently that does not work. And there is no way to make it work (other than creating an issue). Thats definetly bad because all my teams CI will break in that scenario.

Either one needs to be able to override the requies of MyPackages tool_requres.
Or tool_requires is assumed to be a application with its libraries all "private" and not used by the consuming package in any way. In that case the dependency graph should not check if all requirements of my tool requirements have the same version. Only the sub-graphs one level above the consuming package should be checked. For my example graph that would mean that only protobuf and qt are checked if there is any problem.
I would prefer the first solution since the second solution my come with another variant of the same problem.

[Psy-Kai]

The documentation says that

Tool requirements can also be transitive. They can declare their own requirements, both normal requirements and their own build requirements. Normal logic for dependency graph resolution applies, such as conflict resolution and dependency overriding.

So my first suggested solution to this issue is actually the way it should work? I am a bit confused...

[skalde]

we encounter the same problem, in a cross-compile szenario

Example from our Root-Script:

    def build_requirements(self):
        if tools.cross_building(self.settings):
            self.tool_requires("protobuf/3.19.1@gitlab/ci")
            self.tool_requires("grpc/1.45.1@gitlab/ci")

    def requirements(self):
        self.requires("protobuf/3.19.1@gitlab/ci")
        self.requires("grpc/1.45.1@gitlab/ci")

results in following Error:

ERROR: Conflict in grpc/1.45.1@gitlab/ci:
    'grpc/1.45.1@gitlab/ci' requires 'protobuf/3.20.0@gitlab/ci' while 'conanfile.py (sample/None)' requires 'protobuf/3.19.1@gitlab/ci'.
    To fix this conflict you need to override the package 'protobuf' in your root package.

grpc was built against protobuf/3.20.0 in our private repo.

Package reuse as tool_requires seems impossible if dependency resolution is different in build- and host-context. Is there a way out, without introducing an option for each protobuf-version in grpc?

[samuel-emrys]

I note that this has been somewhat addressed in conan 2 by https://github.com/conan-io/conan/pull/11194, but I'm still a bit perplexed as to why the dependency tree needs to be resolved to a diamond. In the following case:

zlib/1.2.11       zlib/1.2.12
    ^                  ^
    |                  |
    |     requires     |
    |                  |
   qt              protobuf
    ^                  ^
    |                  |
    |  tool_requires   |
    +--------+---------+
             |
          MyPackage

Can't both versions of zlib be used independently? As far as I can see, neither of these would leak across the boundary of protobuf or qt, at least where executables are being used, so there shouldn't be a need to converge on a single version of zlib. An option to identify the tree of a tool_requires as an independent tree not to be resolved against other tool_requires would be nice.

Can you provide any insight here @memsharded ?

[raphael-grimm]

@memsharded We got the same problem samuel-emrys is describing.

According to the doc:

tool_requires are designed for packaging tools, utilities that only run at build-time, but are not part of the final binary code. Anything that is linked into consumer packages like all type of libraries (header only, static, shared) most likely are not tool_requires but regular requires.

So treating the trees as independent makes sense. The only problem i see:

The only exception would be testing libraries and frameworks, as long as the tests are not included in the final package.

Here we do not want to treat the tree as independent, since we need the subtree to build our tests.

Hence i think it would be best to add an independent flag to tool_requires (or something of this effect). In this case the user could decide what the right thing to do is.