Help with CI refactoring for conan 2.x

[jeremy-coulon]

I was thinking about how to improve our CI, follow best practices and prepare for conan 2.0. Then I saw this warning on CPT home page:

The project Conan Package Tools does not support Conan 2.x and there is no current planned support.

We are using conan to build our C++ software. We depend on a dozen of 3rd parties, all built with conan and stored in our internal Artifactory. We are not using CPT for our own software development but we use it for all of our dependencies.

CPT is great for building the matrix of all our supported profiles. Currently this is:

• windows, linux
• x86_64, armv8
• debug, release, relwithdebinfo
• gcc 8, 9
• clang 8
• vs2019
• ASan, TSan

Most of our dependencies are coming from conan-center-index. However we cannot use prebuilt binaries from cci because we need to deploy our software to RedHat 7 (glibc 2.17). Prebuilt binaries from cci don't work on this platform.

Thus we have a fork a conan-center-index in which we add a GitLab CI config file and a CPT build.py for each recipe that we need. We try to make as few changes as possible to actual cci recipes in order to be able to merge github/cci into our own fork.

Basically our gitlab CI is trying to replicate what is done in github CI of conan-center-index. AFAIK the internals of this github CI are not open source.

So we are trying our best to build all recipes that we need (but no more), in all the configurations while trying to avoid rebuilding a package that was already built previously.

We also have other in-house conan recipes but conan-center-index is by far the most complex task for us.

Our cci fork has a few drawbacks:

• We have only one Artifactory repository in which we publish packages under testing channel and then promote packages to stable channel. AFAIK this is bad practice and conan 2.0 will remove conan copy command.
• We have to reverse engineer recipe dependencies in order to manually write our .gitlab-ci.yml
• CI takes a long time because we need to start a job for each platform of each package, in which we launch CPT. Some jobs run in parallel but since there are dependencies between packages we end up with 9 sequential steps. Even with CONAN_BUILD_POLICY=outdated this is taking too long.

I was recently thinking about how to improve our CI and thought that I could try to use conan lockfiles in order to figure out which recipes are outdated and to generate CI jobs in the correct order. Am I correct that this is the good way to go?

CPT was great in managing our quite big configuration matrix. But now that it is deprecated I am wondering what should we use instead.

I guess that some tool generating a matrix of build config which creates lockfiles and bundles could be useful in my case.

Sorry for the rather long and fuzzy explanation of my needs.

I can provide samples of my CPT build.py scripts or .gitlab-ci.yml if you need more details.

[uilianries]

Thank you for your feedback! BTW Conan team does not use CPT at all! Including the internal CI, which is pure Jenkins. We will discuss internally about and bring some insight.

[offa]

Are there any news on this topic?

[uilianries]

@offa like it's explained in README, there is no plan so far. The idea is understanding each case and then thinking about how to solve it with Conan 2.x. If you have a need, please, open a new issue explaining your case.

[jeremy-coulon]

Well, i'm still waiting for you to give us some insight. 😇