[question] Please, help to use "build only" fixed version of dependency

apyvovar commented 1 month ago

What is your question?

Hello. Please, help with the following problem, we've encountered.

Prerequisites:

all packages contain binaries only. Sources are not exported with package, so it's impossible to rebuild dependency.
package ID mode for all packages: major

Here is the situation:

What happens here is that we have MyAPP, which uses both libraries - FOO and BAR. FOO requires any BAR in range.

Step 1

Later, BAR/1.1 is introduced, but it's possible, that something may be wrong with it or we just don't want to rely on fresh version of it. For now, we consider it unsafe, until tested properly with our package.

So, now, while creating FOO/1.1 we would like to build it with BAR/1.0 (not the latest in range), but still declare, that FOO/1.1 should be binary compatible with future minor releases of BAR/1.X. What I'm looking for is some kind of syntax to fix version of requirement during build, but to declare binary compatibility of some range downstream. Is pseudo-code, probably, something like this:

def requirements(self):
    self.build_with("BAR/1.0#<pkg_id>") # exact version only to build with, only for this package, invisible downstream

    self.requires("BAR/[>=1.0 <2.0]") # downstream users may require any of BAR/1.X, conan should not report version error

At this point, MyApp uses FOO/1.1, and it's like BAR/1.1 was ignored, even though it's available and latest in range.

Step 2

Finally, later, BAR/1.1 is considered safe (or BAR/1.1.X is introduced), and MyApp now wants to require it. It should be ok with FOO/1.1. Also, this must not trigger rebuild of any of the packages in graph (see above: sources of dependencies are unavailable).

In the end, the result situation should be like this:

Please, help me to achieve this somehow, because it's the main workflow in our company and the solution of such porblems is the very reason why we chose conan in the first place.

Sorry for the long text :) Thank you in advance.

Have you read the CONTRIBUTING guide?

[x] I've read the CONTRIBUTING guide

memsharded commented 1 month ago

Hi @apyvovar

Aren't you looking for a lockfile? I think the above flow can be easily implemented with:

conan lock add --requires=bar/1.0. This will create a conan.lock file with that version
conan create/install ... --lockfile-partial By default will use the conan.lock, but you can do it explicit with conan create/install .. . --lockfile=conan.lock --lockfile-partial

Note the --lockfile-partial is necessary as the lockfile is not complete, it only locks at this moment the bar/1.0, but not the rest of the dependencies, that will be "free" to resolve.

memsharded commented 1 month ago

Also there are other strategies, more artificial without using lockfiles. For example if you remove all bar version, and only install the one you want with something like conan download bar/1.0 in the current cache, then the following commands will prioritize that bar/1.0 version, not 1.1 (just don't pass the --update argument, and it will prioritize the current cache version).

Still, it seems the lockfile is the recommended solution, it was designed for use cases like this one.

apyvovar commented 1 month ago

Hi, @memsharded

I haven't tried lockfiles yet, because it looked quite complex and I thought automatization of it on CI would be error-prone. I'll try to figure out something, and give it a try.

Still, IMO, it would be quite convenient to add some simple "locking" functionality directly in conanfile.

Thank you.

memsharded commented 1 month ago

Still, IMO, it would be quite convenient to add some simple "locking" functionality directly in conanfile.

There are mechanisms already in the conanfile, like defining the version that you want with an override=True or force=True. Furthermore the conanfile can also explicitly depend on a specific recipe-revision. Still this wouldn't be recommended to do for locking purposes, when there is a specific built-in mechanism designed for that.

I haven't tried lockfiles yet, because it looked quite complex and I thought automatization of it on CI would be error-prone.

This is actually the opposite. By definition, it is impossible to do builds in a distributed CI (for example when building for different platforms such as Windows and Linux, using different computers, physical or virtual), with some degree of concurrency that would be consistent without using lockfiles. It is easy for example that someone publish a new version of the dependencies in the middle of a build, and some Windows builds use that newer dependencies and the Linux builds use older dependencies because they were faster and built earlier.

So lockfiles were more complex in Conan 1, but in Conan 2 they have been simplified a lot, and allow to do things like were impossible in Conan 1 (like this partial locking of just 1 dependency). And they are the standard mechanism (beyond Conan) to implement reproducible dependencies versions when there are version ranges, so this would be the general recommended solution.

conan-io / conan