[question] How should `target` info be passed when building a cross-compiler `build_requires` from sources

I'm cross-compiling via build_requires using the new two-profile (build/host) approach and have been following the example in the docs for the compiler recipe:

class CrossCompiler(ConanFile):
    name = "my_compiler"
    settings = "os", "arch", "compiler", "build_type"
    options = {"target": "ANY"}
    default_options = {"shared": False, "target": None}
    def configure(self):
        settings_target = getattr(self, 'settings_target', None)
        if settings_target is None:
            # It is running in 'host', so Conan is compiling this package
            if not self.options.target:
                raise ConanInvalidConfiguration("A value for option 'target' has to be provided")
        else:
            # It is running in 'build' and it is being used as a BR, 'target' can be inferred from settings
            if self.options.target:
                raise ConanInvalidConfiguration("Value for the option 'target' will be computed from settings_target")
            self.options.target = "<target-value>"  # Use 'self.settings_target' to get this value

When building the compiler package I pass the default profile as the host since I want it to run on my build machine: conan create . --profile:host=default

However in order to build the package I need more information. For example, compiler.version is needed to tell me which which sources to fetch. From the snippet above I understood that I need to pass this in from the command-line as an options.target (and if it is being used as a build_requires for another package then it will get this info from the host_profile being used to build that package).

I'm having trouble with how this target option should be passed. I guess it has to contain settings such as arch and compiler.version etc. but I don't know how to pass these as a single option. I expect it is possible since it is in the docs.

Can you please expand on this example? Or perhaps you can point me towards an existing recipe for such a package.

Thanks!

Hi, @GordonJess. I need to say you are living on the edge 🎸 , this feature is still experimental, and using it in the context of a cross-compiler is something we still need to validate. The new approach with the two-profiles has proved to be very robust and solid and it will likely be the default for the next major Conan version... so let's try to work together and improve the example, the documentation and the feature itself.

Creating the cross-compiler package

When building the compiler package I pass the default profile as the host since I want it to run on my build machine: conan create . --profile:host=default

In order to activate the new feature you need to pass the build-profile also. It is ok if both are the same:

conan create <cross-compiler> --profile:host=default --profile:build=default

With this command you are building the cross-compiler, so the recipe for the cross-compiler lives in the host context and there are no target_settings. Why there are no target_settings? We could perfectly allow the user to provide a --profile:target=... argument and they can be used in the recipe... but, from my understanding, typically a cross-compiler serves a bunch of targets (at the very minimum Release/Debug). Some cross-compilers targets many different OS versions like the Android-NDK, or some toolchains (osxcross,...) should work for many different target profiles.

In that sense, maybe the option target only make sense for some cross-compilers, but not for all of them. Take into account that different values of the option will compute different package-ids, and hence, different binaries... if all the binaries are the same, you really don't need that option. For example, the binary for the Android NDK is exactly the same for all the possible os.api_level, it makes no sense to add that option (check this draft here: https://github.com/jgsogo/conan-center-index/blob/android_ndk/recipes/android-ndk/all/conanfile.py).

Other cross-compilers might need more information in order to create its own package. I can imagine a cross-compiler that is able to generate binaries for other distribution/arch, it needs the headers and libraries to use while generating binaries for the target architecture, and you need to know much more information about the future target to fetch the proper sources/binaries. How much information? A full Conan profile (os, arch, compiler, build_type)? Only the basic information (os, arch)?

....that's the reason why I think that the option target is the best approach: some cross-compilers don't need it at all, others will need only the os+arch and other might need a triplet of maybe all the settings. I think this is something specific for each crosscompiler, we cannot generalize and force the user to provide a full --profile:target when it is not needed.

Using the cross-compiler

And then we arrive at the other side of the equation when we are using the cross-compiler as a build-requires:

conan create <library> --profile:host=host --profile:build=default

Here the cross-compiler lives in the build context (defined by the --profile:build=default profile) and the --profile:host=host is what we were referring to as the target before, now for sure we have (and we need) a full profile.

Conan needs to retrieve the proper cross-compiler package, so we need to provide the same settings (the profile --profile:build=default) and also the same value for the target option (if used). And here we really have a problem, two alternatives:

the user provides the option in the command line (or it is hardcoded in the default profile). The first alternative is not very UX-friendly (conan create ..... --options:build=android-ndk:target=api-level-21), and the second one makes no sense in some scenarios:
```
[settings]
os=Linux
arch=x86_64
...

[options]
android-ndk:target=api-level-21
```
I say it makes no sense, because the profile default is the one you use for "everything", and the option related to Android only makes sense in very specific scenarios.
the value of the option is hardcoded into the host profile. This makes a lot of sense, since that profile will likely list the cross-compiler:
```
[settings]
os=Android
os.api_level=21

[options]
android-ndk:target=api-level-21

[build_requires]
android-ndk/version
```
...the only problem is that android-ndk:target=21 is not going to work, that option applies to the host context and not to the build one where the android-ndk binary lives.

We need to figure out a way to pass options from one context to another, or maybe list options that applies to build-requires, for example:

[settings]
os=Android
os.api_level=21

[build_options]  # Options that apply to build context
android-ndk:target=api-level-21

[build_requires]
android-ndk/version

As you can see (I hope I've explained well enough) the new model with the two profiles is sound, but it requires some additional functionality to be feature complete. There are other issues about this same topic (https://github.com/conan-io/conan/issues/6971), and I really appreciate all of you trying to make this work and providing so much valuable feedback. Our plan is to implement as much as possible (and experiment) in Conan v1.xx with these features while they are experimental, and offer stable implementation in Conan v2.0.

I would really like to hear other alternatives or ideas, I'm trying to gather feedback from many different places before making a proposal.

Hope this helps you a little bit. I'm sorry I cannot provide a final answer 😞

Hi @jgsogo!

Thanks for your detailed explanation! It's very helpful for my understanding.

Though I'm not sure that I completely understand the downside to providing target settings through a profile (--profile:target) as long as it is possible to decide which of the settings should have an influence on the package_id.

For example, in my case, I'm packaging the TASKING VX-toolset which includes several compilers for different core architectures. The arch setting does not affect the package binary at all, it is only used in package_info() to set information such as environmental variables (PATH, CC, CXX, etc), include paths etc. So I'd need this kind of control to make sure a different package_id isn't generated for each one:

def package_id(self):
    del self.info.settings_target.arch

So for this reason I don't think options.target works so well in my case.

...how would you propose to pass multiple values (os,arch) through this single option anyway? Or would you make an option for each one (options.target_os, options.target_arch)?

Actually, I realised I can stop options from affecting the package_id by simply removing them in configure() after grabbing their value.

from conans import ConanFile, tools
from conans.errors import ConanInvalidConfiguration
import os, re

class ConanRecipe(ConanFile):
    name = "TASKING_VX-toolset"
    version = "1.0.0"
    author = "Gordon Jess"
    homepage = "https://www.tasking.com/support/tricore-and-aurix-toolset-support"
    url = "https://<scm_server>/repos/compilers/TASKING_VX-toolset"
    description = "TASKING Tricore and Aurix toolset"
    topics = ("tasking", "compiler", "tools")
    settings = { "os", "compiler", "arch" }
    options = { "target_compiler_version": "ANY", "target_arch": "ANY" }
    default_options = { "target_compiler_version": None, "target_arch": None }
    requires = "cmake_toolchain_tasking/1.0.0@tools/master"
    _target_compiler_version = ""
    _target_arch = ""

    def configure(self):
        settings_target = getattr(self, 'settings_target', None)
        if settings_target is None:  # this package is being built ('host' context)

            if self.settings.get_safe("os") != "Windows":
                raise ConanInvalidConfiguration("This package is currently only available for Windows")

            if self.options.target_compiler_version:
                self._target_compiler_version = self.options.target_compiler_version
            else:
                raise ConanInvalidConfiguration("A value for option 'compiler_version' must be provided")

            if self.options.target_arch:
                self._target_arch = self.options.target_arch
            else:
                raise ConanInvalidConfiguration("A value for option 'target_arch' must be provided")

        else:  # this package is being used as a build_requires ('build' context)

            # 'target_' options can be inferred from settings
            self._target_compiler_version = self.settings_target.get_safe("compiler.version")
            self._target_arch = self.settings_target.get_safe("arch")

        # remove options which shouldn't affect package_id
        self.options.remove("target_arch")

    def source(self):
        git_user = tools.get_env("GIT_USER")
        git_pass = tools.get_env("GIT_PASS")
        git = tools.Git(folder="TASKING_TriCore-VX", username=git_user, password=git_pass)
        git.clone("https://<scm server>/a/compiler/TASKING_TriCore-VX/%s" % self._target_compiler_version, "master", shallow=True)

    def package(self):
        self.copy("*", keep_path=True)

    def package_info(self):

        # add paths to environment and set vars for build tools
        compiler = "c%s" % self._target_arch
        bin_folder = os.path.join(self.package_folder, compiler, "bin")
        control_program = os.path.join(bin_folder, "%s.exe" % compiler)
        archiver_program =  os.path.join(bin_folder, re.sub(r'^c(.*)', r'ar\1.exe', compiler))

        self.env_info.PATH.append(bin_folder)
        self.env_info.CC = control_program
        self.env_info.CXX = control_program
        self.env_info.ASM = control_program
        self.env_info.AR = archiver_program

        self.cpp_info.bindirs.append(bin_folder)
        self.cpp_info.includedirs.append(os.path.join(self.package_folder, compiler, "include"))

It'd be useful to be able to set the version of the cross compiler package in set_version() using the target setting/option.

@jgsogo

We need to figure out a way to pass options from one context to another, or maybe list options that applies to build-requires

Just came across this problem (passing the same profile to --profile:build and --profile:host) and Conan complains that some options that are not defaulted are not set.

The workaround for now is to only set --profile:host when creating toolchain-like packages. But build-context options seems like a promising idea to solve this.

conan-io / conan

[question] How should `target` info be passed when building a cross-compiler `build_requires` from sources #7688

Creating the cross-compiler package

Using the cross-compiler