Open cajus opened 2 years ago
You should be able to override (transitive) requirements: https://docs.conan.io/en/latest/using_packages/conanfile_txt.html#overriding-requirements
[requires] libpng/1.6.37@foo_user/foo_channel
Hmm. Does that help here? I'm not trying to build "my app" with a conanfile.txt, but packages from conan-center-index. So for example:
The latter will fail, because it is not able to resolve the self.requires('zlib/1.2.11')
. It's in foo/dev - and that isn't recognized.
I'm looking for a way to set the "default" channel to be used for conan create
. No idea how I should override the dependencies there without too much mess.
Let me give some context: Initially we conceived the user/channel as something dynamic, something that could change and that could serve for moving packages through different stages. We realized it was a mistake, and that packages should be immutable. This was already proposed and validated for Conan 2.0 in this tribe proposal: https://github.com/conan-io/tribe/pull/16
One of the implications is that user/channel should be used as an immutable reference to a package. If you fork conan-center-index and create your own packages with your own user/channel
, that is ok, but the downstream consumers that depend on them should be modified and updated to require =
those packages with that user/channel. Because it is a different package recipe, that has diverged, could have different (and incompatible) behavior. And if there are no modifications, then the user/channel should probably be respected, to indicate that the recipe is being used as-is from conan-center-index.
Thanks for your answer @memsharded - id didn't notice conan-io/tribe#16 yet.
I'm still not sure if you got me right. It's not about my program's conanfile which requires some packges. They've all a libcurl/1.2.3@foo/bar
style requires. It's about the need to modify the requires of "forked" packages from conan-center. So I've to replace all their requirements from libcurl/123
to libcurl/123@foo/bar
. I've been there for the last years now, but that has a downside: as I've to package for various architectures starting from android over ios/macos to webassembly and more between, I needed to tune packages, as 25% of the used ones did not work properly for all profiles.
Forcing me to modify all these self.requires()
makes it harder to contribute other changes/fixes back upstream. In this case, it would be nice to have some override channel for conan create
s requires()
call, or be able give the one and only configured remote a default channel to be used in this case. I'm not mixing any channels. I'm just forced to keep packages localy.
Alternatively I can drop channels completely here and tweak the stable
vs dev
vs whatever
in artifactory. But that creates an additional source for errors - if a developer does not remove conan-center
from the list of remotes, it will result in bad builds.
Facing exactly the same problem here.
We maintain an internal fork of conan-center-index and we publish our own packages to our internal repo. We publish these packages as <package>/<version>@<company>/stable
. We do this to be 100% sure we're always consuming our packages and nothing leaks from the official conan center.
Having the ability to temporaryly require an "official" package is good because it allows us to quickly test stuff, so having our packages "vendored" is a must so we can differentiate them from the official ones.
Building and publishing with our custom user and channel is not a big deal, but maintaining the recipes is. We must edit each recipe to modify the requires and build_requires accordingly (`self.requires(
I'd love the idea of setting a global user and channel so all of our requirements and build_requirements automatically get the
@<company>/stable
appended.
Hi @ericriff
Thanks for the feedback.
We are aware that adding user/channel in a fork from conan-center-index
is not the best experience.
We had considered in the past the possibility of automatically adding the @<company>/stable
part to references and requires, but I kind of recall still having quite many edge cases that made it not a great feature either.
So we have started to recommend the following to use packages from ConanCenter:
The rationale is the following:
conancenter
remote can be systematically removed, for example with conan config install
. It is not used at all, so it is not possible to get something that leaks from it. Packages are created from the fork always, or "missing"conan create
from the conan-center-index
repo, or add the conan remote add conancenter https://center.conan.io
. We have seen quite a few users succeeding with this approach, do you think it wouldn't work for your case?
OK I think I can live with that.
But (and this is an extremely noob question) how do I create a package with no user and channel on conan 1? We're just starting the migration to conan 2 and I don't want to do both things (migrate to conan 2 and ditch our @company/stable
suffix) at the same time.
conan create recipes/libtool/all/ libtool/2.4.6 -pr:b x86_64_gcc10 -pr:h x86_64_gcc10
ERROR: conanfile didn't specify version
The help still mentions we need user and channel
positional arguments:
path Path to a folder containing a conanfile.py or to a recipe file e.g., my_folder/conanfile.py
reference user/channel, version@user/channel or pkg/version@user/channel (if name or version declared in conanfile.py, they should match)
I can just put the @
after the version and nothing more but it looks weird. Is this the expected workflow?
conan create recipes/libtool/all/ libtool/2.4.6@ -pr:b x86_64_gcc10 -pr:h x86_64_gcc10
<builds>
Yes.
conan create . pkg/version@
# and for packages containing name but not version:
conan create . version@
Is the Conan 1.X necessary syntax.
That was one of the reasons Conan 2.0 decided to move to a new CLI like conan create . --version=xxx
, which is way cleaner and better interface.
I think the use case for user/channel is still required for conan-center-index and asking people not to use those is not helping.
Imagine the case where I have a product using a component from conan-center-index and that product has 2 release streams. Using the user/channel would enable me to create a channel per release stream and ensure that a change in the 2nd channel is not impacting the 1st.
This, in many companies, is important, as we cannot hold up on a change until all the potential combinations have been well tested, and having a change potentially impact all releases is very risky.
There are also changes to those packages that do not impact the package itself, i.e. Change to support Conan v2 would not impact the package version itself, yet need to be reflected somewhere (user or channel).
So while for most situations the user/channel may not be worth it, there are still important use cases where they are needed, which makes this scenario important to be enhanced.
From reading the page above, the real recommendation appears to be using "package revision" via the use of lock file. While this is ok, it feels as if the use of a channel per release stream is more user-friendly and can also map to a "branch" that is also very likely to be the model use where each channel use a different conan-center-index branch as to support a multiple release version system.
Using package revision may also make it harder to understand which revision to keep on the cache server on cleanup activity, where using channel, one may decide to only keep the 5 latest revision "per channel".
I think the \
I would imagine the \
The end results should still be a \
I fail to see a value to a \name>/\<version>@\<user/\
In the same way that in git, you either use a branch name or a commit ID, the commit ID is the Conan package revision where the \
This would imply that if you "fork" a package and build it with a different \
Imagine the case where I have a product using a component from conan-center-index and that product has 2 release streams. Using the user/channel would enable me to create a channel per release stream and ensure that a change in the 2nd channel is not impacting the 1st.
This, in many companies, is important, as we cannot hold up on a change until all the potential combinations have been well tested, and having a change potentially impact all releases is very risky.
I totally agree on that. This is very important. But the way to approach this issue is not with user/channel
, the recommended way is using different server repos. This has been a recommended best practice in package management and devops for long time.
Basically the idea is that the new packages that are to be tested/under development you don't put them in the same server repo as the other packages, exactly to not disrupt the other devs and CI jobs builds. So you upload them to a secondary, lets call it "testing" repo. Only when the packages are ok, everything is working, etc, those packages can be promoted (copied) from "testing" to the "main" repo to make them available to other devs and CI jobs.
One critical piece of this flow to work correctly is immutability. Packages cannot be changed, renamed or rebuild in this process, they have to be "promoted" as they were tested, otherwise the previous testing can be completely invalid. For example, it is easy that some recipe does a conditional over if self.channel == "testing"
to do something, and then your production binaries will be untested, because the tested ones were different ones.
There are also changes to those packages that do not impact the package itself, i.e. Change to support Conan v2 would not impact the package version itself, yet need to be reflected somewhere (user or channel).
So while for most situations the user/channel may not be worth it, there are still important use cases where they are needed, which makes this scenario important to be enhanced.
From reading the page above, the real recommendation appears to be using "package revision" via the use of lock file. While this is ok, it feels as if the use of a channel per release stream is more user-friendly and can also map to a "branch" that is also very likely to be the model use where each channel use a different conan-center-index branch as to support a multiple release version system.
Not really, mostly the opposite. Actually having more than 1 "package revision" for the same package version + recipe revision + package_id should be considered a process error, and indicates something went wrong in the CI, or there were unnecessary builds just wasting CI to rebuild something that was already existing with the exact same input sources and configurations.
Moreover, Conan 2 no longer locks package revisions into lockfiles, they just store the package version + recipe revision, that is, they lock down to the "sources" origin, because that, together with the configurations are enough to produce fully deterministic dependency graphs.
I totally agree on that. This is very important. But the way to approach this issue is not with
user/channel
, the recommended way is using different server repos. This has been a recommended best practice in package management and devops for long time.
I think it is ok for some companies to manage that using multiple repo but that is extra work/maintenance, which I would not want to have to deal with unless required. We do use multiple repos (over multiple instances) but only for distribution purposes, not for SDLC aspect. The "release" of something is where we "copy" an artifact from our internal repo to an external repo, and that has nothing to do with the user/channel, for us that is "another" dimension on top of the rest. The repo/server enable us to have internal release and external release. Yet this is not part of the SDLC cycle in the sense that R&D control the SDLC cycle and our Product group control the external release cycle.
Basically the idea is that the new packages that are to be tested/under development you don't put them in the same server repo as the other packages, exactly to not disrupt the other devs and CI jobs builds. So you upload them to a secondary, lets call it "testing" repo. Only when the packages are ok, everything is working, etc, those packages can be promoted (copied) from "testing" to the "main" repo to make them available to other devs and CI jobs.
We expect the opposite, we expect our internal release on the channel "devel" to impact our developer, the idea is for them to pick up the change that was released, we also use locked \
For us a release on our "devel" channel has a version identifying it as such, we normally use -alpha pre-release tag, when we promote we actually generate a new build, and we use a official version (that is the only change allowed during the internal release process), QA then test the official release and when this is good enough we promote to a different repo for deployment to wider audience (that is what your promotion is about).
Internal release always impact the same audience so when we go from devel to master we use different version system to make this visible and we don't just copy the artifact, we change its public version (from a pre-release version to an official one).
For us we need to be able to say "pick the latest artifact from channel A" or "pick this exact version/revision" (doesn't matter which channel because different channel guaranteed different revision due to how we handle the versioning).
One critical piece of this flow to work correctly is immutability. Packages cannot be changed, renamed or rebuild in this process, they have to be "promoted" as they were tested, otherwise the previous testing can be completely invalid. For example, it is easy that some recipe does a conditional over
if self.channel == "testing"
to do something, and then your production binaries will be untested, because the tested ones were different ones.
I do not think immutability has anything to do with the flow. Immutability is about giving a version information you get always the same artifact, the flow you use is irrelevant.
The fact that you rebuild something doesn't invalidate the testing you did on it, as always it depends on what you change in between those builds. In our case, the testing we do on the "devel" channel is not the same as the "master" channel. We do internal team testing on "devel" and the "mainline" is released to QA, they test that and then release it publicly. Yet for us the use of the two channels is important, first we do not want to see the version increase for no value, meaning that during developer using the pre-release versioning such as -alpha.
When we our testing and developer is ok with sending stuff to QA we create an official version and send it off out of our "master" stream and continue working on our "devel" stream. We do want component on devel to be able to use component build either on devel or master as long as themselves are on devel, but if they are on master they must use only official version in their dependencies.
So we have this nice system in place already, and there was inverment made there so unlikely we will change that.
For us the conan-center-index is addressed using script for now, we would certainly appreciate removing that logic from our maintenance cycle and have it officially supported by Conan instead.
Not really, mostly the opposite. Actually having more than 1 "package revision" for the same package version + recipe revision + package_id should be considered a process error, and indicates something went wrong in the CI, or there were unnecessary builds just wasting CI to rebuild something that was already existing with the exact same input sources and configurations.
I disagree with you here, when you change a third-party package you do not control their version, and renaming their package name or changing the version is going to lead to problems. I.e. I need to make a patch to openssl 3.0.12, I cannot change openssl to myproject-openssl as this adds a lot more complication, I need to find all the requires statement on openssl and now point them to mine. Nor can I increase 3.0.12 to 3.0.13 because there may be a community release next using that version. The only thing left is the revision.
To me the revision is really the thing that brings immutability. The rest is mostly decoration. With revision I am certain that if you add a patch or change an existing patch the result will produce a new revision and won't impact anything I had using a explicit reference to the previous revision
I think it is ok for some companies to manage that using multiple repo but that is extra work/maintenance, which I would not want to have to deal with unless required.
This is surprising to hear. Most typically, having separate repositories for the SDLC not only means less work and maintenance with easier to set up permissions and things like retain policies, but specially because of this, it also helps a lot to reduce some costs.
I disagree with you here, when you change a third-party package you do not control their version, and renaming their package name or changing the version is going to lead to problems. I.e. I need to make a patch to openssl 3.0.12, I cannot change openssl to myproject-openssl as this adds a lot more complication, I need to find all the requires statement on openssl and now point them to mine. Nor can I increase 3.0.12 to 3.0.13 because there may be a community release next using that version. The only thing left is the revision.
I think we are mixing here recipe-revisions
and package-revisions
. They are very different things, with very different behaviors. I was talking about package-revisions
, but it seems you are talking about recipe-revisions
.
@memsharded To me a version, name, and revision are always related to a package in the generic sense not in the Conan's definition.
In Conan a package is represented by a recipe, it can then produce "binary caches", yet when we talk about dependencies we talk about packages generically and not about the caching system.
A caching system is normally always optional when missing things still work but take a long amount of time, so the binary cache is not relevant here.
I.e. I cannot say my package depends on this other package's exact binary cache entries. I have to use a name, version and revision and conan then will figure out if the it has a cached binary meeting those requirements.
Correct me if wrong, but that is so far the view I could make out of the documentation and usage.
What you say is correct. But it doesn't invalidate what I explained above:
pkg/version#recipe_revision:package_id
is a model/flow/process error, and should be considered as such when it happens. Lockfiles will not support locking different package-revisionspkg/version
is expected. Lockfiles works locking down to the recipe-revision.When I was responding to:
From reading the page above, the real recommendation appears to be using "package revision" via the use of lock file.
Which was not correct, that is not the recommendation at all, because it cannot be even done.
Ok so I think you misread me then, I will restate my point using the same nomenclature you used as to make things clearer.
I never suggested having multiple package revision for the same pkg/version#recipe_version. What is the package_id you are referring to? Can someone set that as part of their dependencies statement? The package_id seems to be the binary cache entry, to me that is not usable by the human, only by conan as a way to associate binary cache to a recipe.
In your nomenclature, this statement " From reading the page above, the real recommendation appears to be using "package revision" via the use of lock file." should read:
"From reading the page above, the real recommendation appears to be using "recipe revision" via the use of lock file."
I've the scenario that we've a local conan package repository (mostly because of reducing traffic) which keeps the packages. These packages are based on conan-center packages, but may have small modifications/patches for certain architectures.
Let's say a conan recipe does a
self.requires("libpng/1.6.37")
somewhere, this fails, because conan-center is not there as a remote.CONAN_USERNAME
andCONAN_CHANNEL
are deprecated. There's somedefault_user
anddefault_channel
, but I can't get it running from outside the recipe. Looks like I don't understand how to use these :confused:What do I need, to make it pick the package from a certain channel (i.e. @foo/stable) on
conan create
, without replacing all requires with a full qualified pkg/version@channel scheme? Or let it pick it's requirements from the same channel I'm running "create" for?