[question] how to add build requirements to lockfiles (>=1.28)

[GordonJess]

I am following the example for Lockfiles in CI, which is looking great btw, only it does not mention anything about build_requires.

When I am deriving a full lockfile for a product from a base lockfile, my profile contains the cmake_installer package as a build_requirement. After computing the build order and firing the parallel build commands I receive the error:

ERROR: Build-require 'cmake_installer' cannot be found in lockfile

Previously, using the conan graph lock commands I forced this using the --build flag, but this doesn't seem to have any effect.

Lockfile contents:

{
 "graph_lock": {
  "nodes": {
   "0": {
    "ref": "libb/0.2@Concepts/master#43be2d605cc1ac02c3f26bdae24c5efa",
    "options": "",
    "package_id": "75e2bb3a81d61edfcf0ca163cd990739e9f78bcb",
    "prev": "cb6564ee8a379e607da543cd20e6ff94",
    "modified": true,
    "requires": [
     "1"
    ],
    "context": "host"
   },
   "1": {
    "ref": "liba/0.1@Concepts/master#c960eb1771be6c094332069fb511f9c7",
    "options": "",
    "package_id": "5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9",
    "prev": "0eca0aa023769d2d7fe96c492357ce0b",
    "modified": true,
    "context": "host"
   },
   "3": {
    "ref": "app1/0.1@Concepts/master#44b74fb44dfa512b82bfc6190a51f4f5",
    "options": "",
    "package_id": "Package_ID_unknown",
    "requires": [
     "4"
    ],
    "build_requires": [
     "7"
    ],
    "context": "host"
   },
   "4": {
    "ref": "libd/0.1@Concepts/master#385c7ef15cbad2fe4901a2b60bd3cacd",
    "options": "",
    "package_id": "9ba64c59cab1df520e9686c9055b27fc2f2f5ebc",
    "requires": [
     "0",
     "5"
    ],
    "context": "host"
   },
   "5": {
    "ref": "libc/0.1@Concepts/master#9ec9b5aad5bafc8c4bfac176f3acc947",
    "options": "",
    "package_id": "75e2bb3a81d61edfcf0ca163cd990739e9f78bcb",
    "prev": "10d3b2f1a30c82a95389699d02cc8f17",
    "modified": true,
    "requires": [
     "1"
    ],
    "context": "host"
   },
   "7": {
    "ref": "cmake_installer/3.15.3@conan/stable#0",
    "options": "",
    "package_id": "60b04a80ac4dd5c23f1acffacc679e9902e636e5",
    "prev": "0",
    "context": "host"
   }
  },
  "revisions_enabled": true
 },
 "version": "0.4",
 "profile_host": "[settings]\nnarch=tricore\narch_build=x86_64\nbuild_type=Debug\ncompiler=tasking-tricore\ncompiler.flavor=ctc\ncompiler.version=v6.2r2p2\nos_build=Windows\n[options]\n[build_requires]\n*: cmake_installer/3.15.3@conan/stable\n[env]\n"
}

[memsharded]

Hi @GordonJess

Sorry, update from my response, I didn't read it correctly.

I am not sure what could be happening, it might be that the build-require is dropped at some step. I would need to check the intermediate steps and what is being obtained as a result of the build processes. One of the things that changed is that now, the UI is more explicit, it is necessary to use --lockfile-out arg to get the modified lockfile with the result of the build.

Could you please provide something that I could check, including the build-order and install/create commands, and what lockfiles are being generated at each step? If you prefer you could zip something and attach, and I can check here. Many thanks!

[GordonJess]

Hi @memsharded,

I think maybe I used the --build flag at the wrong place before...

I have now added it when deriving the configuration-specific lockfile from the base:

conan lock create ../conanfile.py --user=Concepts --channel=master --lockfile=locks/libb_base.lock --lockfile-out=locks/libb_cfg1.lock --profile=my_profile --build

This avoids the error message from the issue description however the build_requires (although already existing in the cache from when the initial package was created) is output in the calculated build-order:

C:\ci_test\libb\build> conan lock build-order locks/app1_cfg0.lock --json=build_order/app1_cfg0.json 
[[('cmake_installer/3.15.3@conan/stable#0', '60b04a80ac4dd5c23f1acffacc679e9902e636e5', 'host', '10')], [('liba/0.1@Concepts/master#c960eb1771be6c094332069fb511f9c7', '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9', 'host', '1')], [('libb/0.2@Concepts/master#a8db729478ee8cdf09eded1a4e0b8884', 'Package_ID_unknown', 'host', '0'), ('libc/0.1@Concepts/master#9ec9b5aad5bafc8c4bfac176f3acc947', 'Package_ID_unknown', 'host', '5')], [('libd/0.1@Concepts/master#385c7ef15cbad2fe4901a2b60bd3cacd', 'Package_ID_unknown', 'host', '4')], [('app1/0.1@Concepts/master#44b74fb44dfa512b82bfc6190a51f4f5', 'Package_ID_unknown', 'host', '3')]]

Any ideas how to avoid this unnecessary rebuilding?

Thanks, I'll try send a .zip later on!

[GordonJess]

Hi @memsharded,

Here is my modified example with build requires. You will see that cmake_installer/3.15.3@conan/stable is rebuilt from sources as a result of the build-order output.

ci.zip

Thanks!

[memsharded]

Hi @GordonJess

Yes, this behavior is expected: The --build has exactly the same semantics as if you do a conan install --build which means build everything in source. In fact, it tries to achieve the exact same result, but with immutability guarantees that allow to distribute a build and be sure that dependencies don't change.

So if you don't want to rebuild from sources the cmake_installer package, then you need to provide the desired build policy to the conan lock create (typically is a combination of package_id_mode and --build=missing), to define what needs to be built from sources and what not. Please try this approach and let me know.

[GordonJess]

Hi @memsharded.

Ah, got it! Thanks! --build=missing when creating the lockfile works.

[memsharded]

Excellent, very happy that it is working.

which is looking great btw

Good! I was quite happy about the final result, I think it is a good step forward for lockfiles. Looking forward to hearing more feedback and reported issues if necessary to keep improving. Thanks!