search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.23k stars 980 forks source link

[question] How to use a predefined user/channel as "standard" for require() #10441

Open cajus opened 2 years ago

cajus commented 2 years ago

I've the scenario that we've a local conan package repository (mostly because of reducing traffic) which keeps the packages. These packages are based on conan-center packages, but may have small modifications/patches for certain architectures.

Let's say a conan recipe does a self.requires("libpng/1.6.37") somewhere, this fails, because conan-center is not there as a remote. CONAN_USERNAME and CONAN_CHANNEL are deprecated. There's some default_user and default_channel, but I can't get it running from outside the recipe. Looks like I don't understand how to use these :confused:

What do I need, to make it pick the package from a certain channel (i.e. @foo/stable) on conan create, without replacing all requires with a full qualified pkg/version@channel scheme? Or let it pick it's requirements from the same channel I'm running "create" for?

iiknd commented 2 years ago

You should be able to override (transitive) requirements: https://docs.conan.io/en/latest/using_packages/conanfile_txt.html#overriding-requirements

[requires] libpng/1.6.37@foo_user/foo_channel

cajus commented 2 years ago

Hmm. Does that help here? I'm not trying to build "my app" with a conanfile.txt, but packages from conan-center-index. So for example:

The latter will fail, because it is not able to resolve the self.requires('zlib/1.2.11'). It's in foo/dev - and that isn't recognized.

I'm looking for a way to set the "default" channel to be used for conan create. No idea how I should override the dependencies there without too much mess.

memsharded commented 2 years ago

Let me give some context: Initially we conceived the user/channel as something dynamic, something that could change and that could serve for moving packages through different stages. We realized it was a mistake, and that packages should be immutable. This was already proposed and validated for Conan 2.0 in this tribe proposal: https://github.com/conan-io/tribe/pull/16

One of the implications is that user/channel should be used as an immutable reference to a package. If you fork conan-center-index and create your own packages with your own user/channel, that is ok, but the downstream consumers that depend on them should be modified and updated to require = those packages with that user/channel. Because it is a different package recipe, that has diverged, could have different (and incompatible) behavior. And if there are no modifications, then the user/channel should probably be respected, to indicate that the recipe is being used as-is from conan-center-index.

cajus commented 2 years ago

Thanks for your answer @memsharded - id didn't notice conan-io/tribe#16 yet.

I'm still not sure if you got me right. It's not about my program's conanfile which requires some packges. They've all a libcurl/1.2.3@foo/bar style requires. It's about the need to modify the requires of "forked" packages from conan-center. So I've to replace all their requirements from libcurl/123 to libcurl/123@foo/bar. I've been there for the last years now, but that has a downside: as I've to package for various architectures starting from android over ios/macos to webassembly and more between, I needed to tune packages, as 25% of the used ones did not work properly for all profiles.

Forcing me to modify all these self.requires() makes it harder to contribute other changes/fixes back upstream. In this case, it would be nice to have some override channel for conan creates requires() call, or be able give the one and only configured remote a default channel to be used in this case. I'm not mixing any channels. I'm just forced to keep packages localy.

Alternatively I can drop channels completely here and tweak the stable vs dev vs whatever in artifactory. But that creates an additional source for errors - if a developer does not remove conan-center from the list of remotes, it will result in bad builds.

ericriff commented 1 year ago

Facing exactly the same problem here.

We maintain an internal fork of conan-center-index and we publish our own packages to our internal repo. We publish these packages as <package>/<version>@<company>/stable. We do this to be 100% sure we're always consuming our packages and nothing leaks from the official conan center.

Having the ability to temporaryly require an "official" package is good because it allows us to quickly test stuff, so having our packages "vendored" is a must so we can differentiate them from the official ones.

Building and publishing with our custom user and channel is not a big deal, but maintaining the recipes is. We must edit each recipe to modify the requires and build_requires accordingly (`self.requires(/) -> self.requires(package>/<version>@<company/stable) . This is a huge friction point for us, particularly when we sync our fork and there are a lot of merge conflicts because of this.

I'd love the idea of setting a global user and channel so all of our requirements and build_requirements automatically get the @<company>/stable appended.

memsharded commented 1 year ago

Hi @ericriff

Thanks for the feedback.

We are aware that adding user/channel in a fork from conan-center-index is not the best experience.

We had considered in the past the possibility of automatically adding the @<company>/stable part to references and requires, but I kind of recall still having quite many edge cases that made it not a great feature either.

So we have started to recommend the following to use packages from ConanCenter:

The rationale is the following:

We have seen quite a few users succeeding with this approach, do you think it wouldn't work for your case?

ericriff commented 12 months ago

OK I think I can live with that.

But (and this is an extremely noob question) how do I create a package with no user and channel on conan 1? We're just starting the migration to conan 2 and I don't want to do both things (migrate to conan 2 and ditch our @company/stable suffix) at the same time.

conan create recipes/libtool/all/ libtool/2.4.6 -pr:b x86_64_gcc10 -pr:h x86_64_gcc10  
ERROR: conanfile didn't specify version

The help still mentions we need user and channel

positional arguments:
  path                  Path to a folder containing a conanfile.py or to a recipe file e.g., my_folder/conanfile.py
  reference             user/channel, version@user/channel or pkg/version@user/channel (if name or version declared in conanfile.py, they should match)

I can just put the @ after the version and nothing more but it looks weird. Is this the expected workflow?

conan create recipes/libtool/all/ libtool/2.4.6@ -pr:b x86_64_gcc10 -pr:h x86_64_gcc10
<builds>
memsharded commented 12 months ago

Yes.

conan create . pkg/version@
# and for packages containing name but not version:
conan create . version@

Is the Conan 1.X necessary syntax. That was one of the reasons Conan 2.0 decided to move to a new CLI like conan create . --version=xxx, which is way cleaner and better interface.

hpe-ykoehler commented 4 months ago

I think the use case for user/channel is still required for conan-center-index and asking people not to use those is not helping.

Imagine the case where I have a product using a component from conan-center-index and that product has 2 release streams. Using the user/channel would enable me to create a channel per release stream and ensure that a change in the 2nd channel is not impacting the 1st.

This, in many companies, is important, as we cannot hold up on a change until all the potential combinations have been well tested, and having a change potentially impact all releases is very risky.

There are also changes to those packages that do not impact the package itself, i.e. Change to support Conan v2 would not impact the package version itself, yet need to be reflected somewhere (user or channel).

So while for most situations the user/channel may not be worth it, there are still important use cases where they are needed, which makes this scenario important to be enhanced.

From reading the page above, the real recommendation appears to be using "package revision" via the use of lock file. While this is ok, it feels as if the use of a channel per release stream is more user-friendly and can also map to a "branch" that is also very likely to be the model use where each channel use a different conan-center-index branch as to support a multiple release version system.

Using package revision may also make it harder to understand which revision to keep on the cache server on cleanup activity, where using channel, one may decide to only keep the 5 latest revision "per channel".

hpe-ykoehler commented 4 months ago

I think the \/\ and package immutability are not inconsistent with each other...

I would imagine the \ being like a URL to a git repo, whereas the \ is more like a branch of that repo.

The end results should still be a \/\#\ and stored as that.

I fail to see a value to a \name>/\<version>@\<user/\#\ format, it should be either a specific revision or a \/\ but not both.

In the same way that in git, you either use a branch name or a commit ID, the commit ID is the Conan package revision where the \/\ is the branch, that gets you to the "latest" revision (whatever that is at that time).

This would imply that if you "fork" a package and build it with a different \/\ it would still produce the same package \, until you make a change somewhere.

memsharded commented 4 months ago

Imagine the case where I have a product using a component from conan-center-index and that product has 2 release streams. Using the user/channel would enable me to create a channel per release stream and ensure that a change in the 2nd channel is not impacting the 1st.

This, in many companies, is important, as we cannot hold up on a change until all the potential combinations have been well tested, and having a change potentially impact all releases is very risky.

I totally agree on that. This is very important. But the way to approach this issue is not with user/channel, the recommended way is using different server repos. This has been a recommended best practice in package management and devops for long time.

Basically the idea is that the new packages that are to be tested/under development you don't put them in the same server repo as the other packages, exactly to not disrupt the other devs and CI jobs builds. So you upload them to a secondary, lets call it "testing" repo. Only when the packages are ok, everything is working, etc, those packages can be promoted (copied) from "testing" to the "main" repo to make them available to other devs and CI jobs.

One critical piece of this flow to work correctly is immutability. Packages cannot be changed, renamed or rebuild in this process, they have to be "promoted" as they were tested, otherwise the previous testing can be completely invalid. For example, it is easy that some recipe does a conditional over if self.channel == "testing" to do something, and then your production binaries will be untested, because the tested ones were different ones.

There are also changes to those packages that do not impact the package itself, i.e. Change to support Conan v2 would not impact the package version itself, yet need to be reflected somewhere (user or channel).

So while for most situations the user/channel may not be worth it, there are still important use cases where they are needed, which makes this scenario important to be enhanced.

From reading the page above, the real recommendation appears to be using "package revision" via the use of lock file. While this is ok, it feels as if the use of a channel per release stream is more user-friendly and can also map to a "branch" that is also very likely to be the model use where each channel use a different conan-center-index branch as to support a multiple release version system.

Not really, mostly the opposite. Actually having more than 1 "package revision" for the same package version + recipe revision + package_id should be considered a process error, and indicates something went wrong in the CI, or there were unnecessary builds just wasting CI to rebuild something that was already existing with the exact same input sources and configurations.

Moreover, Conan 2 no longer locks package revisions into lockfiles, they just store the package version + recipe revision, that is, they lock down to the "sources" origin, because that, together with the configurations are enough to produce fully deterministic dependency graphs.

hpe-ykoehler commented 4 months ago

I totally agree on that. This is very important. But the way to approach this issue is not with user/channel, the recommended way is using different server repos. This has been a recommended best practice in package management and devops for long time.

I think it is ok for some companies to manage that using multiple repo but that is extra work/maintenance, which I would not want to have to deal with unless required. We do use multiple repos (over multiple instances) but only for distribution purposes, not for SDLC aspect. The "release" of something is where we "copy" an artifact from our internal repo to an external repo, and that has nothing to do with the user/channel, for us that is "another" dimension on top of the rest. The repo/server enable us to have internal release and external release. Yet this is not part of the SDLC cycle in the sense that R&D control the SDLC cycle and our Product group control the external release cycle.

Basically the idea is that the new packages that are to be tested/under development you don't put them in the same server repo as the other packages, exactly to not disrupt the other devs and CI jobs builds. So you upload them to a secondary, lets call it "testing" repo. Only when the packages are ok, everything is working, etc, those packages can be promoted (copied) from "testing" to the "main" repo to make them available to other devs and CI jobs.

We expect the opposite, we expect our internal release on the channel "devel" to impact our developer, the idea is for them to pick up the change that was released, we also use locked \ (include \ if the developer need to lock more precisely) to ensure developer control what they pick up.

For us a release on our "devel" channel has a version identifying it as such, we normally use -alpha pre-release tag, when we promote we actually generate a new build, and we use a official version (that is the only change allowed during the internal release process), QA then test the official release and when this is good enough we promote to a different repo for deployment to wider audience (that is what your promotion is about).

Internal release always impact the same audience so when we go from devel to master we use different version system to make this visible and we don't just copy the artifact, we change its public version (from a pre-release version to an official one).

For us we need to be able to say "pick the latest artifact from channel A" or "pick this exact version/revision" (doesn't matter which channel because different channel guaranteed different revision due to how we handle the versioning).

One critical piece of this flow to work correctly is immutability. Packages cannot be changed, renamed or rebuild in this process, they have to be "promoted" as they were tested, otherwise the previous testing can be completely invalid. For example, it is easy that some recipe does a conditional over if self.channel == "testing" to do something, and then your production binaries will be untested, because the tested ones were different ones.

I do not think immutability has anything to do with the flow. Immutability is about giving a version information you get always the same artifact, the flow you use is irrelevant.

The fact that you rebuild something doesn't invalidate the testing you did on it, as always it depends on what you change in between those builds. In our case, the testing we do on the "devel" channel is not the same as the "master" channel. We do internal team testing on "devel" and the "mainline" is released to QA, they test that and then release it publicly. Yet for us the use of the two channels is important, first we do not want to see the version increase for no value, meaning that during developer using the pre-release versioning such as -alpha. enable us to keep version change in check.

When we our testing and developer is ok with sending stuff to QA we create an official version and send it off out of our "master" stream and continue working on our "devel" stream. We do want component on devel to be able to use component build either on devel or master as long as themselves are on devel, but if they are on master they must use only official version in their dependencies.

So we have this nice system in place already, and there was inverment made there so unlikely we will change that.

For us the conan-center-index is addressed using script for now, we would certainly appreciate removing that logic from our maintenance cycle and have it officially supported by Conan instead.

Not really, mostly the opposite. Actually having more than 1 "package revision" for the same package version + recipe revision + package_id should be considered a process error, and indicates something went wrong in the CI, or there were unnecessary builds just wasting CI to rebuild something that was already existing with the exact same input sources and configurations.

I disagree with you here, when you change a third-party package you do not control their version, and renaming their package name or changing the version is going to lead to problems. I.e. I need to make a patch to openssl 3.0.12, I cannot change openssl to myproject-openssl as this adds a lot more complication, I need to find all the requires statement on openssl and now point them to mine. Nor can I increase 3.0.12 to 3.0.13 because there may be a community release next using that version. The only thing left is the revision.

To me the revision is really the thing that brings immutability. The rest is mostly decoration. With revision I am certain that if you add a patch or change an existing patch the result will produce a new revision and won't impact anything I had using a explicit reference to the previous revision

memsharded commented 4 months ago

I think it is ok for some companies to manage that using multiple repo but that is extra work/maintenance, which I would not want to have to deal with unless required.

This is surprising to hear. Most typically, having separate repositories for the SDLC not only means less work and maintenance with easier to set up permissions and things like retain policies, but specially because of this, it also helps a lot to reduce some costs.

I disagree with you here, when you change a third-party package you do not control their version, and renaming their package name or changing the version is going to lead to problems. I.e. I need to make a patch to openssl 3.0.12, I cannot change openssl to myproject-openssl as this adds a lot more complication, I need to find all the requires statement on openssl and now point them to mine. Nor can I increase 3.0.12 to 3.0.13 because there may be a community release next using that version. The only thing left is the revision.

I think we are mixing here recipe-revisions and package-revisions. They are very different things, with very different behaviors. I was talking about package-revisions, but it seems you are talking about recipe-revisions.

hpe-ykoehler commented 4 months ago

@memsharded To me a version, name, and revision are always related to a package in the generic sense not in the Conan's definition.

In Conan a package is represented by a recipe, it can then produce "binary caches", yet when we talk about dependencies we talk about packages generically and not about the caching system.

A caching system is normally always optional when missing things still work but take a long amount of time, so the binary cache is not relevant here.

I.e. I cannot say my package depends on this other package's exact binary cache entries. I have to use a name, version and revision and conan then will figure out if the it has a cached binary meeting those requirements.

Correct me if wrong, but that is so far the view I could make out of the documentation and usage.

memsharded commented 4 months ago

What you say is correct. But it doesn't invalidate what I explained above:

When I was responding to:

From reading the page above, the real recommendation appears to be using "package revision" via the use of lock file.

Which was not correct, that is not the recommendation at all, because it cannot be even done.

hpe-ykoehler commented 4 months ago

Ok so I think you misread me then, I will restate my point using the same nomenclature you used as to make things clearer.

I never suggested having multiple package revision for the same pkg/version#recipe_version. What is the package_id you are referring to? Can someone set that as part of their dependencies statement? The package_id seems to be the binary cache entry, to me that is not usable by the human, only by conan as a way to associate binary cache to a recipe.

hpe-ykoehler commented 4 months ago

In your nomenclature, this statement " From reading the page above, the real recommendation appears to be using "package revision" via the use of lock file." should read:

"From reading the page above, the real recommendation appears to be using "recipe revision" via the use of lock file."