Best way to promote packages

[MichaelHCline]

As we use pipelines to build packages and then run tests, we thought we would use an Artifactory repo promotion scheme to "upgrade" the status of our packages. For example, we build and create a package on a dev branch, and upload its binaries to conan-dev. Then spend time running through system tests. Once the package is vetted (could take weeks), we would move the package from a conan-dev repo to a conan-release repo in Artifactory. The separate repos has an advantage of we can setup separate purge policies on each repo. (packages will only stay in conan-dev for 3 months, but will stay forever in conan-release). It looks like from you documentation you never talk about moving packages or package promotion like this, but instead you may use a strategy of changing the channel reference of a package, so that name/version@user/dev will get renamed to name/version@user/release, but the package would stay in the same artifactory repo.

1) Do you have a recommendation for the best strategy or industry standard for promotion of packages? 2) If indeed you want to move a package to a different repo, is the best mechanism to do a conan download then conan upload? 2a) We also have to sneaker net a mass amount of packages to another network and Artifactory. What is the best option to do a mass copy of packages?

[memsharded]

Hi @MichaelHCline

No, you are totally right:

• Package promotions from repo to repo is the way to go (we are introducing this concept in https://docs.conan.io/en/latest/versioning/lockfiles/ci.html#repositories)
• We are deprecating the promotion using channels. Packages will be immutable, this has been approved by the Tribe in https://github.com/conan-io/tribe/pull/16/files (see Theconan copycommand will be removed, as it can mutate a package reference)

We need to start making this more evident in our docs.

Do you have a recommendation for the best strategy or industry standard for promotion of packages?

Yes, the best strategy is what you have described, promotions of immutable packages from one server repo to another server repo.

If indeed you want to move a package to a different repo, is the best mechanism to do a conan download then conan upload? 2a) We also have to sneaker net a mass amount of packages to another network and Artifactory. What is the best option to do a mass copy of packages?

Artifactory < 7.13 had a bug in the promotion of Conan packages, but if your Artifactory is above that, you should be able to use the "copy" command in the UI or the Artifactory API to promote packages from one repo to the other. This is much faster than the conan download + upload.

We were using the conan download + upload too in ConanCenter, and it took a long time, but since the last upgrade, we are using Artifactory API and the promotion is orders of magnitude faster.

[gsemet]

I use conan copy to test on a temporary location, and then promote to its final destination. I do not want to create 2 repositories each time. Would it be possible to create 2 packages (one to test), and zip the other package (so no conan repository publication) that would be uploaded to our conan repository only on the official "go" (that is, at the very end of our pipeline, we have this promotion job).

This would allow conan to respect the "immutability" principle but still allow me to make available to the outside world only when I want.

# Merge requests:
build -> pkg@user/testing -> validation tests

# pipeline on master
build -> pkg@user/testing -> validation tests -> more intensive tests -> promotion -> pkg@user/stable

I still think this validation tests step is needed even on master. In theory this would have been caught during MR, but in practice, some tests are so expensive we only do them on master.

Some project might only want to promote on a daily or weekly basis:

# Merge requests:
build -> pkg@user/testing -> validation tests

# pipeline on master
build -> pkg@user/testing -> validation tests -> promote to pkg@user/unstable

# weekly pipeline
build -> pkg@user/testing -> validation tests -> promote to pkg@user/unstable -> more intensive tests -> promotion to pkg@user/stable

Some project uses unstable as an intermediate staging area. In the non-mutable concept this would require to have 3 repositories !!!

[memsharded]

Some project uses unstable as an intermediate staging area. In the non-mutable concept this would require to have 3 repositories !!!

Yes, it will require 3 repositories. This is a known best practice, that has been proven to work at scale for other languages and package types. There are many advantages to this approach:

• Many, many times, it is not possible to use local state: if the build has different configurations (Linux, Windows, etc), or if because of the size of the project, multiple build agents are used to build in parallel. The only way to have somewhere the state of the build, the binaries built so far for the different packages in the graph and for the different configurations is to put them in a server repo.
• Contrary to local caches, server repositories implement de-duplication. For the same package that is promoted between the 3 repos, only 1 copy is stored.
• Promotions are way faster, and way more atomic using server promotions, and it is possible to promote complete graphs orders of magnitude faster than any promotion done with the client.
• And the most important: packages will always be correct. The conan copy changing the user or channel while keeping the recipe and binaries is broken for a variety of scenarios, like recipes using conditionals on the channel, packages using a package_id_mode in which the channel is involved, or requirements to packages that have been promoted and didn't correctly propagate the requirement channel.

It is not necessary to create 3 repos everytime for every change, but instead 3 repos can be used for all different changes. The important piece is not to put things in important repos that haven't passed some quality check.

[db4]

Artifactory < 7.13 had a bug in the promotion of Conan packages, but if your Artifactory is above that, you should be able to use the "copy" command in the UI or the Artifactory API to promote packages from one repo to the other. This is much faster than the conan download + upload.

@memsharded API copy is not going to work with Artifactory CE for C/C++, right?

[gsemet]

Hello. I am still investigating on how to be compatible with Conan 2, because promoting packages from user/channel to user/channel is essential to the way we build and make our package available to the rest of the developers.

The way we build, test and release our package might not be the cutting edge state of the art but it is proven scalable and repeatable. However, like you said, some packages with conditional statement on their user/channel is not compatible with the promotion scheme (and they basically broke everything else, so we tend to not do any logic in the user/channel part of a package in a recipe).

Here is basically our workflow:

• developers works in their respective user/channel, and no promotion will actually even happen)
• we still follow this pattern:

  
  # master pipeline
  build on master
  -> pkg@testing/master
  -> validation tests on master
  -> promotion to unstable
      (this can promote n packages logically linked together at the same time)

tag pipeline

build -> pkg@testing/master -> validation tests -> promote to pkg@projectname/unstable -> manual tests -> promotion to pkg@projectname/stable

I think I can live with requesting project teams to create 3 repositories with 3 retentions policies on Artifactory, instead of 1 repository with 3 retentions policies with slightly hard regular expression to write (we clean old package on unstable and testing). Working with 3 repositories actually simplify a lot this maintainance.

[SzBosch]

Artifactory < 7.13 had a bug in the promotion of Conan packages, but if your Artifactory is above that, you should be able to use the "copy" command in the UI or the Artifactory API to promote packages from one repo to the other. This is much faster than the conan download + upload.

We were using the conan download + upload too in ConanCenter, and it took a long time, but since the last upgrade, we are using Artifactory API and the promotion is orders of magnitude faster.

In a project using conan 2 and Artifactory 7.84 we want to copy a conan package from one Artifactory (source) repo to another one (target). There are 2 different scenarios:

1. both repos are on the same Artifactory server
2. the repos are on different Artifactory servers

To not route all the traffic slowly and costly via a client (conan download & updaload) we are looking for possibilities to do a server-side copy. Furthermore we do not want to copy all versions and all revisions of a conan package. Only very specific version and revision. When I talk about "package" I actually mean the set of recipe + package-binaries.

Basically what we are looking for is a command or function which takes as input a lockfile and transfers exactly and only these recipe revisions and package revisions from one Artifactory repo to an other.

Is there any function of command for that? If not any suggestions? (at least for scenario 1)

As far as I understood the Artifactory copy function it can only copy single files or folders but not a specific conan package revision (which is only a subset of the files and folders in the source repo). This function must be per my understanding also create/update the index.json files properly in the target repo.

[memsharded]

Hi @SzBosch and all.

This thread is a bit outdated. Now, the conan-extensions repo contains a conan art:promote command (experimental, but wip) to run promotions, see https://github.com/conan-io/conan-extensions/tree/main/extensions/commands/art. This does a server-side (fast and efficient) copy, using Conan 2 "package lists" as inputs. Package lists can be generated with different commands in Conan 2, can be aggregated (merged), be used as inputs for different commands, etc. It is likely that we will extend that command to be able to do client side (download+upload) copies, for promoting accross different servers too.

We are also explicitly documenting CI flows including promotions, see ongoing work in https://github.com/conan-io/docs/pull/3799.

I'd suggest to close this ticket as responded, and you can create new tickets for any further question based on those resources. Thanks!