[question] Yocto integration via custom generator

[rohel01]

What is your question?

I am making another attempt to cleanly integrate and manage Yocto SDK in a Conan 2.x enabled development workflow. First, I create a tool package for each Yocto SDK I want to manage. Since Yocto SDK can be quite big (several GB) and are not meant to be moved around, such packages are merely stub aimed at modeling Yocto packages in a given build dependency tree.

Yocto SDK provide environment script wich needs to be sourced so configure the build environment for cross-compilation. So, I created a custom generator to properly register that environment script with the build environment. I had to use a generators for to reasons:

1. Yocto environment scripts use extensions that are not supported by Conan out of the box. So I generate a simple wrapper script with an extension expected by Conan.
2. I found no way to register env scripts in the package_info callbacks of the Yocto SDK recipes. It seems they are only meant to update environment variables. It seems only generators have access to the correct API

Then, I created profiles for each cross-compilation target I support through Yocto. These profiles register a tool require dependency to the corresponding Yocto package

[tool_requires]
zcu102/3.1.7

So far, I have found this design quite clean and simple. But, there is still one usability issue I cannot solve yet: users need to manually activate the Yocto generator (in recipes or via command line). I would like this generator to be automatically activated when a profile or one of the Yocto SDK package is involved in the build tree.

Do you know how I could achieve that? I am akso open to changing my design strategy...

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @rohel01

Thanks for your question

I found no way to register env scripts in the package_info callbacks of the Yocto SDK recipes. It seems they are only meant to update environment variables. It seems only generators have access to the correct API

You are right, the package_info() of a dependency can only propagate cpp_info and environment variables, but not a full call to a script in the dependency. This is something that callers need to do (via a generator)

So far, I have found this design quite clean and simple. But, there is still one usability issue I cannot solve yet: users need to manually activate the Yocto generator (in recipes or via command line). I would like this generator to be automatically activated when a profile or one of the Yocto SDK package is involved in the build tree.

Happy that you find the design clean and simple.

But it is also true that there is no mechanism to transparently inject generators into consumers of a package. The consumers need to be explicitly aware and define the generators they want to use. This is because the generators are mostly designed as passive, you use a CMakeToolchain generator because it will generate a conan_toolchain.cmake file that your build will use. Having injected a CMakeToolchain generator from, lets say a cmake/version tool-require would still be useless if the consumer recipe is not aware and they explicitly pass the -DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake argument.

And all build-system generators are passive, the only special generators that are more general would be the VirtualRunEnv and VirtualBuildEnv, because they leverage env-vars which can be injected into users.

If you control the recipes of the consumers, this should be relatively straightforward to add there some code that uses them, like in self.run("....", env=["conanbuild", "<mypath-to-script>"]. Please let me know if this is viable, if not we might try to think about other approaches (I have already been checking if it was possible to add some feature to Conan to support this, but it doens't seem simple so far)

[rohel01]

Thanks for your answer.

(I think) I solved it using a dedicated pre_generate hook: I first look for a sdk package in the dependency tree. If I find one, I add the generator to the recipe generator list.

Example hook code

def pre_generate(conanfile: ConanFile):

    # ...
    sdk_ref = identify_build_sdk(manifest, conanfile)

    # If not SDK detected, do nothing
    if sdk_ref is None:
        conanfile.output.debug("No SDK registered, deactivating SDK generator")
        return

    # This recipe build uses a SDK, so register the sdk generator
    if not conanfile.generators:
        conanfile.generators = list()

    conanfile.output.info(
        "Activating SDK generator to "
        f"support {sdk_ref.name}/{sdk_ref.version} dependency"
    )
    conanfile.generators.append("SdkToolchain")

So far, it is working fine. What do you think?

[memsharded]

So far, it is working fine. What do you think?

I think that was pretty smart 🙂, and I feel a bit ashamed now to not have been able to think of this alternative myself 😅

I don't think there should be any important risk there, looks clean and understandable (not a hacky thing), so I cannot think of a reason it wouldn't work (beyond the details of identify_build_sdk(manifest, conanfile) that I ignore).

Very happy to see that the designed Conan flexibility can help in these advanced cases, even a bit beyond of what the thinking of the maintainers initially had in mind.

[rohel01]

I am asking because most hook examples I have seen so far do not affect the build artefacts themselves. Indeed, the documentation says hooks are a feature intended to extend the Conan functionalities to perform certain orthogonal operations (emphasis mine).

Here, I use a hook to automatically source Yocto env scripts for cross compilation. This is not an orthogonal modification, even though my hook actually triggers when it is consistent with the target profile and the build dependency tree.

[memsharded]

I am asking because most hook examples I have seen so far do not affect the build artefacts themselves. Indeed, the documentation says hooks are a feature intended to extend the Conan functionalities to perform certain orthogonal operations (emphasis mine).

Good point. That is true, that documentation is there because the hooks have been abused to do even build operations, download operations, etc, that should clearly belong to recipes, and that will render the recipes completely invalid without those hooks.

I see your case also requires the hook for the build to succeed, but the approach looks a bit less coupled, because it only injects a generator, it is protected with sdk_ref = identify_build_sdk(manifest, conanfile), it will only apply to some specific builds/profiles, etc., so even if it reads like not fully orthogonal, it is not as bad as the scenarios that such documentation is trying to prevent.

But sure, it is still true that the best solution would be not to need this hook. The hook is smart and I don't see a high risk in it, but it would be better to not need the hook to achieve successful builds. A "good" solution would be explicitly using the generator in the recipes, but it would require modifying the recipes. This could be done in 2 different approaches:

• Using a conditional to the input settings to dynamically append a generator to the existing ones, for example in the configure() method, sort of what you are doing in the hook.
• Add the generator unconditionally to recipes, even with generators = "sdk_generator", and let the generator code itself to do nothing if the build profile is not the expected one.

[rohel01]

Given your comments, I think the benefits of the hook approach in terms of developer experience outweight the drawbacks. I will keep your suggestions in mind if we encounter any issues in the long run. Thanks for your answers!

[memsharded]

Sounds very reasonable, thanks to you for the feedback!