[feature] platform_requires in toolchain package

quentinbinotcarrier commented 1 month ago

Context

I am currently developing Conan recipes that package Yocto SDKs (see #7431) as toolchain packages. These Yocto SDKs include a manifest listing the "system libraries" available on the host system (e.g. openssl, zlib, boost). Managing these libraries is essential for cross-compilation as the set of host libraries can vary between versions. Currently, there is no direct way to configure these libraries in the recipe itself, which would be preferable over managing them separately in both the recipe and the profile.

Feature Proposal

This feature proposes adding a new configuration section (e.g., system_conf or platform_conf) to toolchain packages, allowing system libraries and their paths to be specified directly within the package. This configuration would work similarly to conf_info / cpp_info / provides, providing a centralized way to list platform or system packages that the host system already provides with versioning information (unlike what provides currently do). Additionally, specifying library paths will enable consumers to link against these libraries effectively.

zlib_lib = os.path.join(self.package_folder, "sysroots", "arm-poky-linux-gnueabi", "usr", "lib", "zlib.so")
zlib_include_dir= os.path.join(self.package_folder, "sysroots", "x86_64-pokysdk-linux", "include")
self.system_conf.provides.append("zlib/1.3.1", {
    "libs": zlib_lib,
    "includedirs": [zlib_include_dir]
})

Additional Requirements

The toolchain package should also support compiling for the build machine while keeping the deps-tree to facilitate unit testing, maintaining consistent dependencies while fetching them from Conan Center.

Rationale for the Workflow

We have multiple products and variant that share a common Yocto-based OS with many different control / ui applications running on each of them.
These applications evolve rapidly, with dependencies changing frequently during testing. It seems way easier and faster to use conan to manage this workflow.
While bitbake may be a choice for the final release build, maintaining the same workflow for development and testing within Conan seemed to offer better traceability and reliability.

Have you read the CONTRIBUTING guide?

[X] I've read the CONTRIBUTING guide

memsharded commented 1 month ago

Hi @quentinbinotcarrier

Thanks for your feedback.

There is an existing Conan feature intended for this use case. It works like this:

Create zlib/system, boost/system (or zlib/1.3.1@myyocto for versioned, or the like) recipes that are wrappers for the yocto installed libraries
Use the [replace_requires] feature in the profile that is used for the yocto build to define which dependencies in the dependency graph will be used, instead of the standard ones

What you are proposing is equivalent to implement all of that in a single toolchain recipe. But this would be more challenging as tool_requires cannot influence the dependency graph, they are typically loaded after resolving the normal "requirements" for regular libraries, so it would be too late to change how the dependency graph depends on some packages like zlib. So the viable approach is by providing the information in profiles as the above [replace_requires] does.

This approach (you can read about replace-requires for example in https://blog.conan.io/2022/01/17/New-conan-release-1-44.html) has several advantages:

No need to modify existing recipes. This is very important for scale
The ability of easily supporting different platforms, with different system libraries, scaling horizontally by providing new pkg/system recipes
Better model when libs need something more than includes/lib-names, like some libs might need some preprocessor definitions, the Conan recipe can provide it.
Total flexibility to decide which libraries comes from where. If at some point some library is not desired from yocto, it can be easily configured at the profile level, without changing recipes again.

So it is less centralized, but it has more flexibility and power. And it is already available in Conan, so this would be the recommended approach at this moment. As commented above, trying to define it in toolchain recipes would have very big architectural challenges, so I don't think this can be considered at the moment.

Thanks very much for your feedback!

quentinbinotcarrier commented 1 month ago

That makes sense, thanks a lot for that quick answer. I am still a little bit concerned using this mechanism since it does not allow proper versioning of that "grouped" replace_requires so that I can resolve it as a dependency.

Would there be a way to have a centralized / version replace_requires? Or maybe to have some kind of profile "package" so that we know we would need to update to a newer profile to get the latest lib for exemple?

memsharded commented 1 month ago

Would there be a way to have a centralized / version replace_requires? Or maybe to have some kind of profile "package" so that we know we would need to update to a newer profile to get the latest lib for exemple?

Yes, there are a couple of machanisms that this can be done:

Have the [replace_requires] defined in the default profile that you are using for a specific yocto toolchain.
Manage profiles with in-profile inclusion include(default) to add the profile you want or by command line composition with --profile=profile_base --profile=profile_special

This way it is relatively easy to have the definitions of the [replace_requires] for a specific toolchain package in a place, then the possible variations you want to manage will include or compose over this base profile, something like:

[settings]
....

[tool_requires]
my_yocto_toolchain/1.0

[replace_requires]
zlib/*: zlib/1.3.1@myyocto

quentinbinotcarrier commented 3 weeks ago

Thanks a lot for you advices, I should be able to do what I want with those mechanisms.

conan-io / conan