[question] Resolve latest compatible set of versions for literal CI/CD

[domfellner]

Hi all,

I’m about to implement a literal CI/CD pipeline, so every commit (alternatively: every release) of any library should trigger a rebuild of the consumer application. We’re using SemVer and therefore model all requirements with version ranges like libA/[^1.0.0].

What I want to accomplish is that the latest comprehensive set of compatible versions is always used for this integration build. So let’s say, my app1 requires libB and libC which both require libA (so we have a classic diamond). The integration pipeline calls ‘conan create’ on app1, possibly as reference with the lockfile of the triggering package.

My naïve approach was to set app1’s requirements to libB/[*] and libC/[*]. This works as long as all latest versions are compatible (let’s assume all libs are 1.0.0 for now, requiring [^1.0.0] versions respectively). If libA releases a 2.0.0, app1 will still be buildable and ignore the new libA (as no new libB or libC is available), but if libB releases a 2.0.0 (requiring libA/[^2.0.0]), the diamond cannot be resolved any more. As there cannot be a fully integrated version of app1 with the new version (libC/2.0.0 is missing), the integration build will fail with a version conflict which is expected at this point.

When libC releases a 1.1.0 version afterwards, still relying on libA/[^1.0.0], there could be a fully compatible set of packages consisting of libA/1.0.0, libB/1.0.0, libC/1.1.0 and app1/latest. With app1 requiring the latest versions, it will always choose libB/2.0.0 and there is an error, but I want to fall back to the compatible set.

I could adapt app1’s requirements manually for different sets of versions, but following a literal CI/CD approach, this should be resolved automatically.

I’m thinking of some option in the version like

requires = [ 
    “libB/[*, latest_compatible=True]",
    "libC/[*, latest_compatible=True]"
]

which will start with the latest versions and in case there is a conflict will fall back and check the next previous version etc. The order of preference is given in the order of requirements (first requirements shall be downgraded at last). I’m aware that this can be time consuming, but would be an optional feature and I’d be willing to pay for it.

I wonder if some more people have the same use case. Even with lockfiles I couldn’t get this working. Am I missing something?

Thanks a lot for all the good work! Conan is coming to stay.

Regards Dom

• [x] I've read the CONTRIBUTING guide.

[domfellner]

Am I the only one with this use case? Are there some initial thoughts how to approach this?

[memsharded]

Hi @domfellner

Yes, this is a complicated problem, let me summarize some important points:

• The conan dependency resolution is not complete, it doesn't implement the necessary SAT solver to find the joint compatible solution to a set of version ranges.
• It doesn't do it, because such is a NP-hard problem. While the algorithm could still be implemented, the problem is that it will not resolve graphs in a reasonable time in practical terms.
• The SAT solution doesn't scale because one of the characteristics of Conan (well, actually about C, C++ dependency management) is the very large amount of conditional dependencies: conditional on the os/platform, you can get different dependency graphs, it is also common to find conditional dependencies based on the current versions themselves, and also it is super common to have "feature-switches" based on options (if self.options.with_xxx => require(xxx) )
• So the above would imply a lot of backtracking, and each one would involve fetching files from servers, unzipping them, and evaluating the conanfiles. We have estimated that the resolution times would quickly explode from seconds (current solution) to tenths of minutes, easily.
• The current dependency resolution algorithm is greedy, it starts evaluating from the consumer, which always defines higher priority, and expands upstream, overriding dependencies by name. While not as theoretically powerful, it has proven very effective in practice. The jointly compatible dependencies versions definition relies more on the consumer side than on the algorithm.
• It works relatively well in practice, because some of the above assumptions, that we can make the App depends on libB/[*] and libC/[*] is not valid. You cannot upgrade libB or libC to any arbitrary major version (with breaking changes), and still have App build cleanly, almost by definition it will be broken and it will not build. If the ranges are more aligned like libB/[^1] and libC/[^1], and both 1.X series of LibB and LibC keep a common dependency on libA/[^1], then, everything will fit more easily.
• Conan 2.0 resolution algorithm will improve a bit about using jointly compatible version ranges, trying to find compatible solutions when possible, but it still won't be a full SAT solver.

Please let me know if the above clarifies a bit.

[AbrilRBS]

Closing as answered - While Conan 2 has and likely will introduce improvements over the algorithm, it's not currently expected to implement any kind of complete resolution. Thanks for the question!