[question] Lockfiles without (or with less strict) build requirements

[fschoenm]

What is your question?

The lockfiles by default always lock all dependencies including library dependencies of course but also build dependencies. Now, I can see why this makes sense as it locks all tools (at least Conan-specified tools) to make the build as reproducible as possible.

However, I think in many cases there's no benefit or need to be so strict when it comes to tools involved in the build process. Why should it matter, if some of my build tools were built with cmake/3.26.1 and others with cmake/3.28.2 or different versions of meson or even if some of my build tools dependencies have been updated in the meantime? The output will most likely be the same. When rebuilding everything though, I suddenly need different versions of CMake or Meson or other tools instead of just using the latest (allowed) version of my tools. I also have to keep them around on my remote forever.

Also, some/many tools are not even that strictly version-locked, especially the compiler, which in many cases does not come from a Conan package and usually has the biggest impact of all on the build output.

I know that in theory, I can omit build requirements and use partial lockfiles. However, this is quite a bit of manual work to maintain. Partial lockfiles would also apply to other dependencies and could mask missing lockfile entries.

I wonder what your opinion on the matter is or if you have thought about this. Maybe there could be different modes for partial lockfiles that allows some restrictions, such as allowing missing build requirements. (This would also need to be considered when conan creates/updates lockfiles to include only "normal" dependencies.)

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Thanks for your question @fschoenm

, if some of my build tools were built with cmake/3.26.1 and others with cmake/3.28.2 or different versions of meson or even if some of my build tools dependencies have been updated in the meantime? The output will most likely be the same.

Most likely be the same is important here.

I think that build systems, even those like CMake tht keeps a very good forward compatibility to newer versions, that compatibility might be in some cases "API-compatible", that is, the build will keep working with the same files, and be able to do a build. But sometimes build systems do change their defaults, and might change some policies, default flags, etc., and there is a chance that they actually produce a slightly different binary. Maybe not necessarily binary-incompatible, but maybe affect some performance, or some security/hardening, etc. That is assuming perfect forward API-compatibility, which only happens in a perfect world. Maybe not the fault of build-system maintainers that do a great job, but it is enough that there is some misuse of something in the consumer side build-scripts, and this might break with newer versions of the build system.

So in general, the price to pay keeping a few Kbs for meson packages and some Mbs for cmake packages, for achieving full perfect reproducibility is definitely much better than risking facing some maybe a bit unlikely but not impossible reproducibility issues that might take a very long time to debug.

Also, some/many tools are not even that strictly version-locked, especially the compiler, which in many cases does not come from a Conan package and usually has the biggest impact of all on the build output.

But those are "baked" into the package_id of the packages created with that compiler.version setting, down to the level that the user decides.

[fschoenm]

@memsharded Thank you for the quick reply. Yes I understand, I wouldn't expect that to be the default but an opt-in feature if the user wants to assume that "risk".

But those are "baked" into the package_id of the packages created with that compiler.version setting, down to the level that the user decides.

True but that only defaults to the major version of the compiler.

So in general, the price to pay keeping a few Kbs for meson packages and some Mbs for cmake packages, for achieving full perfect reproducibility

Also true but not even on conan-center is the CMake version strictly enforced: "We generally consider tools like CMake as a standard tool to have installed in your system."

(Our compiler packages are up to multiple GB per version in size by the way.)

My point is that the build requirements in the lockfile are very strictly enforced but the arguably most important build tools are not. I would just like a feature that allows us to opt-in to all the build requirements being maintained manually. I think partial lockfiles already allow for this but only in a very awkward way.

[memsharded]

Also true but not even on conan-center is the CMake version strictly enforced: "We generally consider tools like CMake as a standard tool to have installed in your system."

ConanCenter does not use lockfiles, and has different constraints that a more production/product environment that wants to use lockfiles for future reproducibility.

My point is that the build requirements in the lockfile are very strictly enforced but the arguably most important build tools are not.

It depends on what is the approach, there are Conan users using absolutely everything, including compilers as Conan packages.

In any case, Conan can lock what it can lock, if something is not that modeled, Conan will not lock it, but if the build-requires are well defined, there is no reason to not lock them properly, besides saving a few MBs of storage in the server, something that still looks a very minor advantage compared to the strong guarantees that a lockfile provides.

At the moment unlocking build-requires is something like:

conan lock remove --build-requires="*"
conan install ... --lockfile-partial

with the possibility that this would be hiding some "host" dependency that hasn't been locked. The remediations for this would be:

• Compare in the pipeline the "requires" in the lockfile before and after the conan install, to double check
• A possible --lockfile-partial=build-requires extension that only unlocks the build-requires, but not others

This might be a possible feature, tagging it as such. Not a high priority at the moment, though.

[fschoenm]

It depends on what is the approach, there are Conan users using absolutely everything, including compilers as Conan packages.

I know, we're using Conan like this.

As I said, I'm not arguing for changing the defaults but about an opt-in feature that IMO would make lockfiles more flexible by allowing users to lock only what they want to be locked. The current mode of --lockfile-partial mode seems too heavy-handed as it completely disables error checking.

Thanks for your consideration!