search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.31k stars 986 forks source link

[question] CI pipeline with conan and repackaging with new version #6589

Open TheQwertiest opened 4 years ago

TheQwertiest commented 4 years ago

We are trying to come with a proper way to integrate conan in our CI pipeline.

The current idea is something like this: Note: using a single profile for brevity.

All these steps are (seemingly) doable, except for the last one.

So, the questions :)

  1. Does this workflow conform to the (theoretical) canonical conan way?
  2. Is there a way to perform the last step using conan (that is without manually hacking the published package)?

Thanks in advance!

jgsogo commented 4 years ago

Hi! From the point of view of Conan, any change in the version part of the Conan reference is considered a different package (I think #6191 is a really good place to read about it).

So a way to go would be to store the -rc.1.#build_counter as properties in Artifactory, you upload the name/v1.2.3 Conan package to the remote with a property like rc=True, buildNumber=XXand then your manual steps will modify the values of these properties or remove/change them. That way the promotion (and all the dependants of those binaries that are declared in the build info) would take place only on packages with the proper values of those properties.

Using JFrog CLI is a cool and really powerful way to interact with packages stored in Artifactory, it allows to make really complex queries and execute actions over the selected artifacts. And it can be easily added as a custom step in a CI.

petermbauer commented 4 years ago

We are using the channel here for such "release candidate" packages, e.g. mylib/1.2.3@ci/rc<build number>. Then the final release package can be created as a copy with conan copy.

TheQwertiest commented 4 years ago

@petermbauer , this one of the approaches that we've considered. But it seems to be frowned upon (i.e. considered to be not a true conan way): "We are lately recommending not using the channels for doing promotions, but using different remote repositories." Source: https://github.com/conan-io/conan/issues/6191

puetzk commented 2 years ago

So a way to go would be to store the -rc.1.#build_counter as properties in Artifactory, you upload the name/v1.2.3

But this semes like of wrecks immutability for any developer machine's cache and consumer pacakges. They have a package which claims to be name/v1.2.3, but it's really some pre-release that might not even be the same source as what was finally tagged. And conan will happy keep using it, and thus building further downstream packages against the wrong thing, unless you realize this and manually delete it.

If additional testing passed successfully, promote the package to a release build by republishing package as a v1.2.3 (without additional suffixes) Is there a way to perform the last step using conan (that is without manually hacking the published package)?

When we do a recipe revision to support additional profiles, I will sometimes use conan export-pkg --package-folder, to take the binaries of a previously-built release and re-export them linked to the new recipe (rather than recompiling from source, which wouldn't necessarily produce identical binaries due to compiler non-determinism). Which always feels dirty, but seems like least bad of the current options.

memsharded commented 2 months ago

A new CI tutorial is being added to the docs in https://github.com/conan-io/docs/pull/3799, which will close this ticket. Feedback welcome.