[question] How can I get conan install to check more than the latest recipe revision for a matching package

[marvin-the-mathematician]

What is your question?

I have a custom fork of the conan center index (https://github.com/conan-io/conan-center-index) and to save time I don't build every required package every time I run the custom CI/CD pipeline in that fork, i.e. if I already have an acceptable build of Boost in my custom Conan remote then I don't really want to build it again (on all of the platforms that I currently support) just because I am adding support for a new platform. However, since I am using revision_mode = "scm" in my conanfile.py a new package revision is created (associated with the new git commit hash), which becomes the latest version of the recipe. Unfortunately, this causes downstream users of conan install to fail because a matching package id is not available in the latest revision of the recipe even though a matching package id is available at previous revisions of the same recipe :-(

My question is, is there a way to force conan install to consider recipe revisions other than the latest revision in a given remote? Maybe even consider all of the available recipe revisions; choosing the latest one that matches? Or do I have to build and package everything (on all platforms) each time a new platform is added so that the latest recipe revision always has a matching pre-built package for all of the platforms that I support?

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[marvin-the-mathematician]

The downstream conan install invocation looks like this:

conan install \
      --remote my_remote \
      --requires=boost/1.85.0 \
      --profile:all=linux-amd64-gcc11-release \
      --options:all='boost/*:shared=False'

with the linux-amd64-gcc11-release profile being as follows:

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu20
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux

[options]
boost/*:without_charconv=True
boost/*:without_cobalt=True
boost/*:without_context=True
boost/*:without_contract=True
boost/*:without_coroutine=True
boost/*:without_fiber=True
boost/*:without_json=True
boost/*:without_locale=True
boost/*:without_nowide=True
boost/*:without_program_options=True
boost/*:without_stacktrace=True
boost/*:without_test=True
boost/*:without_timer=True
boost/*:without_type_erasure=True
boost/*:without_url=True
boost/*:without_wave=True

[memsharded]

Hi @marvin-the-mathematician

Thanks for your question

My question is, is there a way to force conan install to consider recipe revisions other than the latest revision in a given remote? Maybe even consider all of the available recipe revisions; choosing the latest one that matches? Or do I have to build and package everything (on all platforms) each time a new platform is added so that the latest recipe revision always has a matching pre-built package for all of the platforms that I support?

No, this would make the problem of resolving the dependency graph NP-hard, in a space where every hypothesis is very costly to evaluate, as it requires downloading a new recipe-revision from the server, unzipping it, loading the python files, evaluating the recipe, expanding the subgraph of dependencies again, evaluating the package_id, to see if a binary exists in the server or not. This would take too many hours or days for relatively normal dependency graphs.

Source code changes creating new recipe revision should build the new binaries necessary belonging to that revision, actively at every change or more lazily, when they are necessary. But backtracking to previous versions containing matching binaries is not possible.

What is possible is to avoid the problem by not creating new recipe revision for every repo change. You might want to use revision_mode = "scm_folder" instead, so you are only getting new recipe revisions when the actual recipe is changing, but not other recipes in the repo, specially when you are using a conan-center-index like repo.

[marvin-the-mathematician]

Thanks for the prompt reply (as always). I see. I think my comment about revision_mode = "scm" may have been a red herring (sorry) since I am not changing the recipes provided by the conan center index fork. But thanks for the comment about the revision_mode = "scm_folder" mode; I will take a look at that.

Since the introduction of support for a new platform isn't something that happens often (I was forced to upgrade from manylinux2014 (based on CentOS 7) to manylinux_2_28 (based on AlmaLinux 8) due to the recent end of life of CentOS 7) I will probably opt for rebuilding for every supported platform in CI if the revision of a given recipe has changed since the last time it was built and uploaded to the remote. Thanks.