[feature] Add ability to specify separate "tools" and "library" repos in Conan 2

[System-Arch]

One of my favorite aspects of Conan is the ability to manage and deploy toolchains (compilers, build tools etc.) in an automated manner as part of using Conan for application/product libraries. This feature has gotten even better in Conan 2.0 with the cleaner distinction between "build" and "host" profile settings and the "tool_requires" profile feature. (As an aside the nomenclature is still somewhat confusing, i.e., tool_requires vs. the older build_requires, which actually aligns better with the "build" profile, but I digress).

I realize that I can upload tools to a separate Conan Artifactory repo by specifying the -r options to "conan upload", but the feature I am looking for is the ability to add a qualifier to each entry in my remotes.json file indicating whether a repo is suitable for resolving "tool" (build profile) vs. "library" (host profile) dependencies. My motivation for this request is to be able to establish a clear demarcation between libraries that may ultimately ship to customers/end users vs. those used only for internal purposes. I may have missed something, but I am not currently aware of any way to provide this separation today.

In summary,, I'm looking for something like: conan remote add conan-tools http://artifactory;8082/artifactory/api/conan/conan-local-tools" --tools [could be the default] conan remote add conan-prod http://artifactory;8082/artifactory/api/conan/conan-local-prod" --notools

Thank you for your consideration and for building such a useful and powerful tool.

• [X] I've read the CONTRIBUTING guide.

[memsharded]

Hi @System-Arch

tinction between "build" and "host" profile settings and the "tool_requires" profile feature. (As an aside the nomenclature is still somewhat confusing, i.e., tool_requires vs. the older build_requires, which actually aligns better with the "build" profile, but I digress).

This was because build_requires, despite all the recommendations, was very abused as "build-time" only libraries, that is, when a shared library was linking against a static library, and that had a few bad side effects, so we renamed it to tool_requires to clearly indicate it is for tools like "cmake", "meson", and other similar tools.

I realize that I can upload tools to a separate Conan Artifactory repo by specifying the -r options to "conan upload", but the feature I am looking for is the ability to add a qualifier to each entry in my remotes.json file indicating whether a repo is suitable for resolving "tool" (build profile) vs. "library" (host profile) dependencies.

I think that what you are requesting "per-remote" is not possible, because there are a few counter examples to this:

• Packages like protobuf that acts both like a library and as a tool
• Packages that are libraries but transitive dependencies of a "tool-require", like cmake->openssl. It would be weird an inconsistent that cmake package cannot be installed only from the "tools" repo, because it cannot find a library that is installed from another different repo.

My motivation for this request is to be able to establish a clear demarcation between libraries that may ultimately ship to customers/end users vs. those used only for internal purposes. I may have missed something, but I am not currently aware of any way to provide this separation today.

Instead there are other different ways to achieve this:

• When you are building things, with install or create you can get a json representation of the graph. That contains the context = build|host for each package, and you can use that in CI to upload different packages to different places. That also addresses the issues above.
• It would be possible to call your tools packages with something_tool/1.0.. or use the channel pkg/1.0@user/tool, then use conan upload *tool* or something like that. Maybe a bit too explicit, but some naming convention might be doable.

But probably it doesn't make sense to separate packages in the "upload" stage in different repos, but keeping the consistency of the graph, I would upload things to one repo. Then as things progress, I would copy (promote) packages from one server repo to another. This is also the recommended way to do CI without breaking other developers. When you finally want to release to the final users, you can run a conan graph info --format=json (this is 2.0 syntax) for example, and based on that you can extract information what packages are in the "host" context. Furthermore, it will tell you which binaries don't need to be extracted, because they are pre-compiled artifacts of consumers (like the shared linking static library). And you have the Conan 2.0 new deployers that can be used to extract artifacts from Conan packages, outside of Conan for final deployment or release to customers by other means and not Conan packages.

[System-Arch]

Hi @memsharded,

Thank you for your detailed and thoughtful reply.

tinction between "build" and "host" profile settings and the "tool_requires" profile feature. (As an aside the nomenclature is still somewhat confusing, i.e., tool_requires vs. the older build_requires, which actually aligns better with the "build" profile, but I digress).

This was because build_requires, despite all the recommendations, was very abused as "build-time" only libraries, that is, when a shared library was linking against a static library, and that had a few bad side effects, so we renamed it to tool_requires to clearly indicate it is for tools like "cmake", "meson", and other similar tools.

My original comment might have been misleading. I like the name "tool_requires" and was actually advocating that the "build" profile could be more descriptively named the "tools" profile as presumable everything running on the "build" host is, in fact, a tool. In this vein, I am happy to see that the Conan 2.0 documentation specifies that the legacy build_requires() method is being replaced with a tools_requires() method

I think that what you are requesting "per-remote" is not possible, because there are a few counter examples to this:

Packages like protobuf that acts both like a library and as a tool Packages that are libraries but transitive dependencies of a "tool-require", like cmake->openssl. It would be weird an inconsistent that cmake package cannot be installed only from the "tools" repo, because it cannot find a library that is installed from another different repo.

The cmake example is actually one of several instances of the issue I am seeking to address--another being gcc's dependency on zlib. I have actually built separate versions of these dependencies (i.e., openssl and zlib) using my "build" (a.k.a. "tools") profile for both build and host so that they are built consistently with how the apps (cmake and gcc) are themselves built (e.g., using the system gcc and static linkage, for example). Then I use my newly built versions of cmake and gcc (in this case gcc 10.3 vs gcc 8, which came with the OS distro) to build my production versions of zlib and openssl, in which case they are built as shared libs along with other production settings such as -rpath, etc. Thus I am indeed seeking the ability to pull all of my tools dependencies (both those specified using the [tools_requires] entries in the host profile as well as any recipe-specific ones specified explicitly via the newly introduced tools_requires( (formerly build_requires) method call from a designated "tools repo". More importantly, I want to ensure that no production dependencies (i.e., self.requires()) are pulled from the tools repo, which may have artifacts that are unsuitable for production use.

Instead there are other different ways to achieve this:

When you are building things, with install or create you can get a json representation of the graph. That contains the context = build|host for each package, and you can use that in CI to upload different packages to different places. That also addresses the issues above. It would be possible to call your tools packages with something_tool/1.0.. or use the channel pkg/1.0@user/tool, then use conan upload tool or something like that. Maybe a bit too explicit, but some naming convention might be doable.

I am already doing something akin to the the first suggestion in that I pass an extra option to my CI tooling when building a library or app as a "tool". This "--tool" option causes two things to happen:

1. I use (as noted above) the "build" (preferable named "tools") profile for both build and host configurations.
2. I can thus easily upload the resulting artifacts to a separate tools repo

So my challenge is how to use this tools repo in resolving dependencies when building production libraries (or other artifacts). I considered the second suggestion as a way to achieve this, but as you note, it would require modifying any CCI recipe that uses the build_requires) (now tools_requires) method to add user/channel info and that seems rather undesirable unless there is some global way to do this that I am unaware of. I am using the [tools_requires] feature of the host profile, and that would be easy to change, but I don't think it would cover the build/tools_requires calls embedded in conanfile.py recipies.

But probably it doesn't make sense to separate packages in the "upload" stage in different repos, but keeping the consistency of the graph,

I think this is where our perspectives have diverged and it's why I was so excited by the potential of Conan 2.0. It almost gives me the ability to maintain what I am hoping for, which is two entirely separate graphs--one for my tools and one for my production libraries akin to what I think Conan is doing to distinguish between build and host graphs. I'm just relabeling them mentally as "tools" and "production" rather than "build" and "host" . I think that taking this perspective to its logical conclusion would make Conan 2.0 even cleaner and easier to understand as our production graph is plenty complicated in its own right so keeping the tools out of it would be a nice simplification. I understand that not all users will need or care about this distinction but the same is true of cross-building, Many don't need that either, but the same underlying infrastructure can meet that need as well as my desired separation of concerns (tools vs. product).

Thanks for hearing me out.