[question] Dependent package is not rebuilt when triggering conan install --build=missing

[NilsKlimanis]

What is your question?

Hi! We are using Conan 2.4.1 with a mixture of packages of different origins, some open source, some are our own libraries. Now we have the following situation: Let's say we have three packages A, B, C and M (leaving out other dependencies, but I think they shouldn't matter) M depends on all packages. C depends on A and B, B depends on A. A is a header-only library, B is a shared library, C is currently a static library. M is the "main" recipe of our product.

We generate a lockfile for M and then basically do a conan install --build=missing when we want to build updated packages. Initially, only package A had a package-type set ("header-library"), the others didn't had any package-type set.

First experiment: When I now change recipe A and trigger conan install --build=missing then only packages A (obviously) and package B are rebuilt. But against my expectation, C is not rebuilt.

Second experiment: I added the appropriate package-types to package B and C and then changed recipe A again. This time A, B, and C were being rebuilt, as expected.

Third experiment: Now I changed recipe B, and triggered the rebuild. But this time, only B was rebuilt, but not C. The conaninfo.txt for package C in the [requires] section contains references to A and B that look like this:

A/x.y@user/channel#***:***
B/x.y@user/channel

This means, that the reference for A contains the revision and package id but B does not.

How can this be explained? I would have thought, that dependent packages are all automatically rebuilt? What could block this?

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @NilsKlimanis

Thanks for your question.

only package A had a package-type set ("header-library"), the others didn't had any package-type set.

This doesn't matter if the packages B, C have the shared option defined. If they have it, they will automatically deduce the package-type. Otherwise, it would be very important to define the package_type to have the right propagation of information.

Second experiment: I added the appropriate package-types to package B and C and then changed recipe A again. This time A, B, and C were being rebuilt, as expected.

Exactly, it is necessary to do it

Third experiment: Now I changed recipe B, and triggered the rebuild. But this time, only B was rebuilt, but not C. The conaninfo.txt for package C in the [requires] section contains references to A and B that look like this:

This is where the versioning becomes very important. Not all changes trigger rebuilds, depending on the package types, that define an "embed-mode" or "non-embed-mode", the rebuilding is triggered by versioning.

When there is "embed-mode" then basically every change in recipes will trigger rebuilds. But when there is "non-embed-mode", then only minor version updates trigger rebuilds, but not patches. This way it is extremely efficient to do a change in a .cpp file of a static library, you can update a dependency graph of dozens of other static library packages depending on this one, and no one will need a rebuild, if you only bump the patch version of that package.

You can read more about it in https://docs.conan.io/2/reference/binary_model/dependencies.html, and also https://docs.conan.io/2/reference/conanfile/attributes.html#reference-conanfile-attributes-package-id-modes might have some interesting further details

[marlamb]

Hi @memsharded ,

I have some follow-up questions. As far as I understand the main reason to use the non-embed mode is to get rid of redundant builds saving compute times. On the down-side this requires quite some detailed control over the software built. I have two examples were I would be interested how they are meant to be. Both are basically targeting the restriction mentioned in the documentation

This assumes that there are not inlined functionalities in the dependency headers, and the headers are pure interface and not implementation.

I could imagine one very common use-case are patches, e.g. to compile libraries with another compiler or newer language standards. Patches applied by at the beginning of the build step are IMO in general a very powerful tool. Now to my understanding the patch should not change the actual version of the packaged software (at least I have not seen something like that in the conan-center-index, even for patches that are changing header files and could therefore even be seen as major), mainly because we should not use potential future release numbers. Does that mean every recipe patched that way should also be flagged with the package_id_non_embed_mode attribute to full_mode? Currently grepping through the conan-center-index there is only one match for package_id_non_embed_mode (which is even setting it the the value it is by default anyhow), which looks pretty optimistic to me.

The second example are templates. As far as I can see many open source libraries are using a mixture of templates and non-templated code. We can use opencv as a prominent and widely-used example. What would be the correct annotation for this kind of library? It isn't header-only, but it contains templated code, such that the restriction for the "non-embed mode" is very difficult to track down (we could of course assume that the maintainers consider every change in a header as a breaking major change, but it would very much surprise me if that would work in reality...). However, I currently assume that the library is either considered a shared-library or a static-library from conan point of view, depending on the options and conan will use that information to deduce if it can use the "non-embed mode", correct? If that is true, what would be the best annotation for such a library to state a mixture of "header" and "non-header" library?

Looking forward to understand a little more about the design decisions and ideas how to use conan most efficiently :smile: .

[memsharded]

Thanks for the feedback @marlamb

I could imagine one very common use-case are patches, e.g. to compile libraries with another compiler or newer language standards. Patches applied by at the beginning of the build step are IMO in general a very powerful tool. Now to my understanding the patch should not change the actual version of the packaged software (at least I have not seen something like that in the conan-center-index, even for patches that are changing header files and could therefore even be seen as major), mainly because we should not use potential future release numbers. Does that mean every recipe patched that way should also be flagged with the package_id_non_embed_mode attribute to full_mode? Currently grepping through the conan-center-index there is only one match for package_id_non_embed_mode (which is even setting it the the value it is by default anyhow), which looks pretty optimistic to me.

There are two very different use cases here:

• ConanCenter and conan-center-index recipes, focused at packaging third party open source packages, in which the version assigned to the package matches 1:1 with a source snapshot (tag, release, commit), and only changes in the recipe happen, but direct changes in the original source code never happen. Recipe revision changes should be related to conanfile.py changes, not to packaging a different tag of the original repo (this will in fact be rejected in ConanCenter).
• When packaging your own packages for your own source code, where your own source code continuously evolve and change, and recipe revision can be caused by source code changes that don't bump the version, as well as conanfile.py recipe changes.

The assumption is that patches in ConanCenter are exclusively or almost exclusively to do the minimal adaptations necessary to build system scripts to allow them build and integrate with other dependencies. Patches to C/C++ source code are very discouraged, and most of them should be rejected nowadays in ConanCenter, unless there are very strong reasons. It is true that in the past it was a bit more relaxed but not anymore. Source changes should be done in the upstream repos, not in ConanCenter recipes. So when doing patches to ConanCenter recipes, the assumption is that they will not be changing the public interfaces of the package, and then the default package_id modes will still be pretty safe.

The second example are templates. As far as I can see many open source libraries are using a mixture of templates and non-templated code. We can use opencv as a prominent and widely-used example. What would be the correct annotation for this kind of library? It isn't header-only, but it contains templated code, such that the restriction for the "non-embed mode" is very difficult to track down (we could of course assume that the maintainers consider every change in a header as a breaking major change, but it would very much surprise me if that would work in reality...). However, I currently assume that the library is either considered a shared-library or a static-library from conan point of view, depending on the options and conan will use that information to deduce if it can use the "non-embed mode", correct? If that is true, what would be the best annotation for such a library to state a mixture of "header" and "non-header" library?

The current package_type = "library" is good, and then let it behave as static library or shared library depending on the options. When there are new changes to opencv source code, that will imply new versions for opencv package, not new revisions. If the project versioning bumps only a patch version but is doing changes to public interfaces, that wouldn't be very semver-compliance. In that case it would be good to change the non-embed-mode to patch_mode instead of the current minor_mode. But so far this hasn't been reported.

As a summary:

• The default in Conan 2 is way more conservative than in Conan 1 what used a default, overly optimistic semver_mode for all cases, including embed and non-embed
• The default in Conan 2 for embed-mode is fine, as it is using down to the package-id and including recipe revision, any change in the dependencies will trigger a re-build, completely safe.
• The default in Conan 2 for non-embed-mode is way more conservative than it was in Conan 1 and it provides the perfect middle ground to give full control to users that can version expressively:
  • bump patch or do not bump (new recipe revision), to not build non-embed dependants (changes to implementation files, but not public headers)
  • bump minor to force the build of non-embed dependants, but expecting them not to break (compatible changes in public header files)
  • bump major if the changes in the public header files are breaking and require code changes in the consumers.

[memsharded]

Any further question @marlamb ?

Maybe we can close the ticket as responded, you can always create new tickets for any new question or issue. Thanks!

[marlamb]

From my point of view sufficiently answered. Perhaps we should also ask @NilsKlimanis if he has any further questions?

[memsharded]

Sure, sorry, I didn't see @NilsKlimanis was the OP