search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.29k stars 982 forks source link

[bug] Conan mixing up setting variables of profiles when doing cross compilation #10947

Closed derived-coder closed 4 months ago

derived-coder commented 2 years ago

I cross compile zlib from x86_64 to armv8 like this:

conan create cci/recipes/zlib/all 1.2.11@ -pr:h gcc-9.0-qemu-armv8-release-toolchain -pr:b gcc-9.0-linux-x86_x64-release

I have created a conan tool chain recipe, which I reference in my profile gcc-9.0-qemu-armv8-release-toolchain

However, when executing the create cmd. I get following output:

Cross-build from 'Linux:x86_64' to 'Linux:armv8'
Installing (downloading, building) binaries...
ERROR: There are invalid packages (packages that cannot exist for this configuration):
yocto-toolchain/1.0.0@company/stable: Invalid ID: os=Linux, distro=None is not supported by yocto-toolchain (no binaries are available)
yocto-toolchain/1.0.0@company/stable: Invalid ID: os=Linux, distro=None is not supported by yocto-toolchain (no binaries are available)

My toolchain recipe has a validate step:

def _settings_distro_supported(self):
        return self.conan_data["sources"][self.version]["os"][str(self.settings.os)].get(str(self.settings.os.distro)) is not None
def validate(self):
        if not self._settings_distro_supported():
            raise ConanInvalidConfiguration(f"os={self.settings.os}, distro={self.settings.os.distro} is not supported by {self.name} (no binaries are available)")

I created a new sub setting in settings.yml called distro. For unknown reason this setting is None during cross compilation, although it shouldn't be.

See here my profiles used:

with following profiles: gcc-9.0-qemu-armv8-release

[settings]
os=Linux
os.distro=custom_distro ## here custom sub settings!!!
arch=armv8
compiler=gcc
compiler.version=9
compiler.libcxx=libstdc++11
build_type=Release
[options]
[build_requires]
[env]

gcc-9.0-qemu-armv8-release-toolchain

include(gcc-9.0-qemu-armv8-release)
[options]
yocto-toolchain:target=armv8
[tool_requires]
yocto-toolchain/1.0.0@company/stable

Build profile: gcc-9.0-linux-x86_x64-release

[settings]
os=Linux
os_build=Linux
arch=x86_64
arch_build=x86_64
compiler=gcc
compiler.version=9
compiler.libcxx=libstdc++11
build_type=Release
[options]
[build_requires]
[env]

Remark here, I do not have the os.distro setting. Not needed IMHO.

So I think conan i using the gcc-9.0-mbient-armv8-release-toolchain with the settings.os from the build profile. Which is wrong, I think. Why is it done like this?

Environment Details (include every applicable attribute)

Conan log from profiles during build

Configuration (profile_host):
[settings]
arch=armv8
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux
os.distro=custom_distro
[options]
yocto-toolchain:target=armv8
[build_requires]
*: yocto-toolchain/1.0.0@company/stable
[env]

Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux
os_build=Linux
[options]
[build_requires]
[env]
memsharded commented 2 years ago

The toolchain recipe lives in the "build" context, and if you are using there the settings, those will be the "build" settings, not the host ones. If you want to reference the target settings you need to use the self.settings_target.

If that is not the case, please put together that in a repo in something we could reproduce, the above is missing a few important pieces, like the exact command you are using, what you have defined in the conandata.yml files, the recipes...

derived-coder commented 2 years ago

Hi @memsharded thanks for looking into it. I uploaded all necessary files.

conan_toolchain_testcase.tar.gz Please follow these instructions:

  1. Copy settings.yml into your conan home dir.
  2. Create toolchain recipe: 
    conan create . 1.0.0@ -pr gcc-9.0-qemu-x86_x64-release  -o yocto-toolchain:target=x86_64
  3. Cross compile zlib with the toolchain profile 
    conan create cci/recipes/zlib/all 1.2.11@ -pr:h gcc-9.0-qemu-armv8-release-toolchain -pr:b gcc-9.0-linux-x86_x64-release

    You should see following output:

    
Exporting package recipe
zlib/1.2.11 exports: File 'conandata.yml' found. Exporting it...
zlib/1.2.11 exports: Copied 1 '.yml' file: conandata.yml
zlib/1.2.11: Calling export_sources()
zlib/1.2.11 export_sources() method: Copied 1 '.txt' file: CMakeLists.txt
zlib/1.2.11 export_sources() method: Copied 2 '.patch' files: 0001-fix-cmake.patch, 0002-gzguts-xcode12-compile-fix.patch
zlib/1.2.11: A new conanfile.py version was exported
zlib/1.2.11: Folder: /home/user/.conan/data/zlib/1.2.11/_/_/export
zlib/1.2.11: Exported revision: 76f8021bd02ad1e211f01faae850e3ab
Configuration (profile_host):
[settings]
arch=armv8
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux
os.distro=my-distro
[options]
yocto-toolchain:target=armv8
[build_requires]
*: yocto-toolchain/1.0.0
[env]

Configuration (profile_build): [settings] arch=x86_64 arch_build=x86_64 build_type=Release compiler=gcc compiler.libcxx=libstdc++11 compiler.version=9 os=Linux os_build=Linux [options] [build_requires] [env]

zlib/1.2.11: Forced build from source zlib/1.2.11 (test package): Installing package Requirements zlib/1.2.11 from local cache - Cache Packages zlib/1.2.11:1705e75affd0cc82fd81b01f1f5bae8be44c6c1b - Build Build requirements yocto-toolchain/1.0.0 from local cache - Cache Build requirements packages yocto-toolchain/1.0.0:INVALID - Invalid

Cross-build from 'Linux:x86_64' to 'Linux:armv8' Installing (downloading, building) binaries... ERROR: There are invalid packages (packages that cannot exist for this configuration): yocto-toolchain/1.0.0: Invalid ID: os=Linux, distro=None is not supported by yocto-toolchain (no binaries are available) yocto-toolchain/1.0.0: Invalid ID: os=Linux, distro=None is not supported by yocto-toolchain (no binaries are available)



PS: I also tried your suggestion, with `self.settings_target` but then I fail at step 2. Creating the recipe.
memsharded commented 2 years ago

Step 2 fails at that step because it is missing the "build" profile, which is necessary for cross compilation. Please use the 2 profiles for all the cross compilation scenarios. Also, if the thing is a toolchain, you might want to add --build-require to indicate that at conan create time.

If that keeps failing, please submit a reduced case just for the Step 2 creation, with the settings_target (.zip appreciated too, thanks)

derived-coder commented 2 years ago

What do you mean it is failing at step 2? It is failing for you at step2? The test case should work as provided. You might only adjust the path to your zlib from cci, in step 3

conan_toolchain_testcase.zip

I slightly adjusted step 2:

conan create . 1.0.0@ -pr gcc-9.0-qemu-x86_x64-release  -o yocto-toolchain:target=armv8

Step 3 failing:

conan create cci/recipes/zlib/all 1.2.11@ -pr:h gcc-9.0-qemu-armv8-release-toolchain -pr:b gcc-9.0-linux-x86_x64-release

Step 3 working:

conan create cci/recipes/zlib/all 1.2.11@ -pr:h gcc-9.0-qemu-armv8-release-toolchain -pr:b gcc-9.0-qemu-x86_x64-release

You see the difference? -pr:b gcc-9.0-qemu-x86_x64-release vs. -pr:b gcc-9.0-linux-x86_x64-release

-pr:b gcc-9.0-qemu-x86_x64-release does contain the os.distro setting, -pr:b gcc-9.0-linux-x86_x64-release does NOT contain the os.distro setting.

I created the toolchain with -pr gcc-9.0-qemu-x86_x64-release. And when now doing a cross compile, I thought it is valid to use a build profile what ever I want to use, in my case -pr:b gcc-9.0-linux-x86_x64-release.

Because in my arm toolchain gcc-9.0-mbient-armv8-release-toolchain I have everything defined correctly for the toolchain recipe, in order to get used.

include(gcc-9.0-qemu-armv8-release)
[options]
yocto-toolchain:target=armv8
[tool_requires]
yocto-toolchain/1.0.0@company/stable

Do you see my problem now? In my host profile everything is correctly declared, settings, options, etc... in order to find the package. Why does the build profile is mixed with consuming the toolchain of the host profile? I hope the problem is now clearer ^^

memsharded commented 2 years ago

Step 2 is still wrong. You are only providing the "host" profile: 

conan create . 1.0.0@ -pr gcc-9.0-qemu-x86_x64-release  -o yocto-toolchain:target=armv8

You need to provide the "build" profile too with --profile:build. Otherwise you cannot use the settings_target, and if you can't use the settings_target it will not work later as expected. If you don't fix it here at step 2 first, step 3 doesn't make sense. Also, adding the --build-require (unless test_package already encodes it as a build_require). Otherwise you are creating the toolchain package wrong and will not be able to use it later succesfully. When using distro that is the distro of the target inside the toolchain package, not the build, not the host.

derived-coder commented 2 years ago

when I create the toolchain recipe like that: conan create yocto-toolchain/all 1.0.0@ -pr:h gcc-9.0-qemu-armv8-release -pr:b gcc-9.0-linux-x86_x64-release -o yocto-toolchain:target=armv8

then I do not have self.settings_target

def validate(self):
    if self.settings_target is None:
        raise ConanInvalidConfiguration("Not invoked with cross compilation, missing: 'settings_target'")
yocto-toolchain/1.0.0: Invalid ID: Not invoked with cross compilation, missing: 'settings_target'

Can you tell me the cmd I should use to create the toolchain and what changes I should do in the toolchain recipe?

PS: I am using the same approach from https://github.com/conan-io/conan-center-index/blob/master/recipes/android-ndk/all/conanfile.py#L33 for my validation, but just added the additional: "distro" setting.

memsharded commented 2 years ago

Yes, you need to add the --build-require, as commented above:

conan create yocto-toolchain/all 1.0.0@ -pr:h gcc-9.0-qemu-armv8-release -pr:b gcc-9.0-linux-x86_x64-release -o yocto-toolchain:target=armv8 --build-require
derived-coder commented 2 years ago

I have to admit, I am completely confused now how conan handles cross compilation.

Anyway

conan create yocto-toolchain/all 1.0.0@ -pr:h gcc-9.0-qemu-armv8-release -pr:b gcc-9.0-linux-x86_x64-release -o yocto-toolchain:target=armv8 --build-require

This will not make self.settings_target available during package creation. I get the same error

More important, I do not know how I need to change the toolchain recipe in order to handle the creation and the usage of the recipe/package.

derived-coder commented 2 years ago

I updated the test case, too simplify it more. Should now work also under windows I hope. And I renamed the profiles to host and build toolchain_test_case_update_2.zip

derived-coder commented 2 years ago

One suggestion. Maybe, instead of helping me out fixing my recipe, it is maybe better to create a template toolchain recipe, which everybody can use for their toolchain? I mean you see the logic it is not much. I am just downloading some binaries and then I am setting environment variables. That what all toolchains/SDK will do. So everybody can reuse it.

derived-coder commented 2 years ago

@memsharded

  1. So I am just using the toolchain recipe as is.
  2. Create the toolchain like that: 
    conan create yocto-toolchain/all 1.0.0@ -pr build

  3. And consume it like that:

    conan create cci/recipes/zlib/all 1.2.11@ -pr:h host -pr:b build

Not sure if this is the "correct" way. Anyway, what do you think about create a general toolchain recipe for the docs?

SSE4 commented 2 years ago

@uilianries @danimtb can you help reviewing yocto toolchain above? I am not familar with it

uilianries commented 2 years ago

@derived-coder please, read https://docs.conan.io/en/latest/systems_cross_building/cross_building.html#conan-v1-24-and-newer very carefully. I see you tried to use the old approach.

Also, toolchains can not have a template, because they are particular implemented, there is no such pattern.

The Yocto project works different, each package is added as a layer, is not same as Conan "philosophy". Have you tried https://github.com/conan-io/meta-conan ?

derived-coder commented 2 years ago

@uilianries Where do you see that I am using the old approach? I basically copied more or less the template from the new approach.

I disagree, toolchains can have template ( I used the same approach for a QNX toolchain recipe, works fine). No more setting up the SDK, keeping the SDK up to date, etc. all thanks to conan profiles. It is working fine. However, I am unsure whether it is really "correct" in conan way. E.,g. I have a problem with the distro setting. When cross compiling a recipe, I also need this distro setting in the build context. I think this is wrong. It should be only necessary in the host context.

No, I have not used this approach yet. I am using a different approach which is documented in the conan docs for yocto. I created a toolchain recipe, in order to very easily creating from a recipe, which is placed in the source reop, a artifact which you can easily copy to the target.

Instead of the yocto approach, having a monolithic build, out of source repo.

memsharded commented 4 months ago

Closing this as staled and outdated, please create new tickets against Conan 2 if necessary, thanks for the feedback.