[question] How to publish mulitple package in parallel?

[barnou-psee]

What is your question?

Hello, in our CI we're building for different profiles and we're uploading them. Each profile is built in parallel and when published it happens that the upload steps doesn't see the recipe uploaded but when uploading it, another profile have done it already. How do you handle those cases? I saw there is a --only-recipe option that can be used to upload only the recipe, is there an opposite? liks --only-package maybe? Is it the objective of the -p option?

Have you read the CONTRIBUTING guide?

• [x] I've read the CONTRIBUTING guide

[memsharded]

Hi @barnou-psee

Thanks for your question.

This is indeed a known issue, we have this ticket to track it: https://github.com/conan-io/conan/issues/16703 As you can see there, it is labeled as "complex:huge", because it is mostly a server-side issue, and that can be challenging and require time.

I saw there is a --only-recipe option that can be used to upload only the recipe, is there an opposite? liks --only-package maybe? Is it the objective of the -p option?

I think you are close to what would solve it. As Conan has checks to avoid uploading things that are already in the server, it is possible to do the following:

• First upload with --only-recipe in a single job, all the relevant recipes. The recipes are the ones that might case race conditions if uploaded concurrently at the very same time
• Then, once the recipes are there, you are good to upload concurrently all the packages. As the recipe is already there, Conan will know it and will not re-upload again, so not producing race conditions.

There could be other possible related approaches:

• use the conan cache save/restore together with Conan package-lists, to gather everything in CI side, then do a single upload with the recipe plus all the binaries, in a single job
• retry the upload when failed. If it is due to race conditions, basically the next retry in a few seconds will manage to work fine. This is not a very elegant approach but works relatively well in practice, we have seen users doing this with success (still some of the above would be recommended, this could be like a quick temporary workaround in existing pipelines, as the above might take a bit more to implement)