Open dornbirndevelops opened 1 year ago
dornbirndevelops
commented
1 year ago the failing LOC I was looking at was introduced
due to this issue: https://github.com/conan-io/conan-center-index/issues/15070 in this PR: https://github.com/conan-io/conan-center-index/pull/15154
does someone else have the same issue? what would be a good solution that does not necessarily break the issue above (👀 @SpaceIm)?
SpaceIm
commented
1 year ago It's not how you should translate yocto env vars to conan profile. CC & CXX are only for compiler path.
Pass:
tools.build:cflags/tools.build:cxxflagstools.build:definestools.build:sharedlinkflags/tools.build:exelinkflagstools.build:sysrootAnd prefer tools.build:compiler_executables to CC/CXX (both can be used).
So something like this:
[settings]
os=Linux
arch=armv8
compiler=gcc
compiler.version=11
compiler.libcxx=libstdc++11
build_type=Release
# for conan v2 recipes
[conf]
tools.build:compiler_executables={"c": "aarch64-poky-linux-gcc", "cpp": "aarch64-poky-linux-g++"}
tools.build:cflags+=["-mcpu=cortex-a53", "-march=armv8-a+crc+crypto", "-fstack-protector-strong", "-O2", "-Wformat", "-Wformat-security", "-Werror=format-security"]
tools.build:cxxflags+=["-mcpu=cortex-a53", "-march=armv8-a+crc+crypto", "-fstack-protector-strong", "-O2", "-Wformat", "-Wformat-security", "-Werror=format-security"]
tools.build:defines+=["_FORTIFY_SOURCE=2"]
tools.build:sysroot="/usr/local/poky-x86_64/sysroots/cortexa53-crypto-poky-linux"
# for conan v2 recipes also, shouldn't be necessary when recipe is properly written to listen [conf]
[buildenv]
CC=aarch64-poky-linux-gcc
CXX=aarch64-poky-linux-g++
CFLAGS="-mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong -O2 -Wformat -Wformat-security Werror=format-security"
CXXFLAGS="-mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong -O2 -Wformat -Wformat-security Werror=format-security"
CPPFLAGS="-D_FORTIFY_SOURCE=2"
SYSROOT="/usr/local/poky-x86_64/sysroots/cortexa53-crypto-poky-linux"
# legacy stuff conan v1 recipes
[env]
CC=aarch64-poky-linux-gcc
CXX=aarch64-poky-linux-g++
CFLAGS="-mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong -O2 -Wformat -Wformat-security Werror=format-security"
CXXFLAGS="-mcpu=cortex-a53 -march=armv8-a+crc+crypto -fstack-protector-strong -O2 -Wformat -Wformat-security Werror=format-security"
CPPFLAGS="-D_FORTIFY_SOURCE=2"
SYSROOT="/usr/local/poky-x86_64/sysroots/cortexa53-crypto-poky-linux"thank you for the quick and detailed response. i will try out your solution and close the issue once it works.
dornbirndevelops
commented
1 year ago CC & CXX are only for compiler path.
not a feasible solution, since the CC/CXX variable can also be more than just the compiler path. Options that are required to make the compiler work correctly can be included
are there other options/opinions to solve this?
SpaceIm
commented
1 year ago You mean that CMake fails while testing your C/C++ compiler or what? Does it really matter if options are isolated in CFLAGS & CXXFLAGS?
dornbirndevelops
commented
1 year ago You mean that CMake fails while testing your C/C++ compiler or what? Does it really matter if options are isolated in CFLAGS & CXXFLAGS?
I mean that pretty much every Yocto SDK does not specify anything wrong in particular. Generally setting compiler flags in the CC/CXX variable is allowed and hence should work also for boost, which it does with such a change: boost/all/conanfile.py:
- contents += f" \"{cxx_fwd_slahes}\""
+ contents += f" {cxx_fwd_slahes}"The reason why I want to discuss this first is because the change above might undo the fix for someone else. 😢
dornbirndevelops
commented
1 year ago FTR: I updated the main description above for easier reproducability.
upsilona
commented
1 year ago I'm facing the same issue, sounds like we are more and more using the Yocto SDK with Conan!
I can confirm your fix is needed for Yocto SDK use case
jwillikers
commented
1 year ago @SpaceIm The Conan profile example you provided doesn't actually work because the Boost recipe doesn't consider the tools.build.sysroot option. I've submitted #17315 to address this, which allows cross-compiling the Boost recipe again.
dornbirndevelops
commented
1 year ago hello @jwillikers, I tested your fix to determine whether or not it would also solve this issue. unfortunately it didn't work out.
warning: toolset gcc initialization:
warning: can not find user-provided command 'aarch64-poky-linux-g++ --sysroot=/usr/local/pokysdk-x86_64/sysroots/cortexa53-crypto-poky-linux'
warning: initialized from /workspaces/product-build-test/.conan/data/boost/1.81.0/_/_/source/src/tools/build/user-config.jam:5
/workspaces/product-build-test/.conan/data/b2/4.9.6/_/_/package/166d2a90346126aadca4204c1d127c5ff2a48bff/bin/.b2/tools/common.jam:1104: in common.find-compiler from module common
error: toolset gcc initialization:
error: provided command '"aarch64-poky-linux-g++ --sysroot=/usr/local/pokysdk-x86_64/sysroots/cortexa53-crypto-poky-linux"' not found
error: initialized from /workspaces/product-build-test/.conan/data/boost/1.81.0/_/_/source/src/tools/build/user-config.jam:5the main difference I can see between the previous issue and this one is that the spaces were in a different place:
e.g. /path/with spaces/to/clang vs. gcc --sysroot=/path/to/sysroot
jwillikers
commented
1 year ago hello @jwillikers, I tested your fix to determine whether or not it would also solve this issue. unfortunately it didn't work out.
warning: toolset gcc initialization: warning: can not find user-provided command 'aarch64-poky-linux-g++ --sysroot=/usr/local/pokysdk-x86_64/sysroots/cortexa53-crypto-poky-linux' warning: initialized from /workspaces/product-build-test/.conan/data/boost/1.81.0/_/_/source/src/tools/build/user-config.jam:5 /workspaces/product-build-test/.conan/data/b2/4.9.6/_/_/package/166d2a90346126aadca4204c1d127c5ff2a48bff/bin/.b2/tools/common.jam:1104: in common.find-compiler from module common error: toolset gcc initialization: error: provided command '"aarch64-poky-linux-g++ --sysroot=/usr/local/pokysdk-x86_64/sysroots/cortexa53-crypto-poky-linux"' not found error: initialized from /workspaces/product-build-test/.conan/data/boost/1.81.0/_/_/source/src/tools/build/user-config.jam:5the main difference I can see between the previous issue and this one is that the spaces were in a different place: e.g.
/path/with spaces/to/clangvs.gcc --sysroot=/path/to/sysroot
@dornbirndevelops I've cross-compiled successfully with Yocto, but in order to do so you'll want to tweak your profile. You can leave CC, CXX, CPP and LD as they are. Add a tools.build.compiler_executables conf value like so:
[conf]
tools.build:compiler_executables={'asm': 'aarch64-poky-linux-as', 'c': 'aarch64-poky-linux-gcc', 'cpp': 'aarch64-poky-linux-g++'}I also prepend the bin directory containing the cross-compilers to PATH in the buildenv section so that Boost can find them.
My profile is heavily templated, but here's that snippet.
[buildenv]
PATH=+(path){{ os.path.join(sdk_build_sysroot, "sysroots", build_compiler_prefix, "usr", "bin", cross_compiler_prefix) }}Here's my complete profile as a reference.
{% set cmake_toolchain_file = os.path.realpath(os.path.join(profile_dir, "..", "..", "toolchains", "yocto-toolchain.cmake")) %}
{% set yocto_sdk_directory = os.path.join("/", "usr", "local", "yocto-poky-sdk") %}
{% set sdk_target_sysroot = os.path.join(yocto_sdk_directory, "sysroots", "cortexa72-crypto-poky-linux") %}
{% set build_compiler_prefix = "x86_64-poky-linux" %}
{% set sdk_build_sysroot = os.path.join(yocto_sdk_directory, "sysroots", build_compiler_prefix) %}
{% set cross_compiler_prefix = "aarch64-poky-linux" %}
{% set architecture_compiler_flags = ["-march=armv8-a+crc+crypto", "-mtune=cortex-a72"] %}
{% set sysroot_compiler_flag = "--sysroot=" + sdk_target_sysroot %}
{% set compiler_flags = " ".join(architecture_compiler_flags) + " " + sysroot_compiler_flag %}
[settings]
os=Linux
arch=armv8
arch.extensions=crc+crypto
arch.processor=cortex-a72
compiler=gcc
compiler.version=11
compiler.libcxx=libstdc++11
compiler.cppstd=20
[env]
PATH=['{{ os.path.join(sdk_build_sysroot, "sysroots", build_compiler_prefix, "usr", "bin", cross_compiler_prefix) }}']
CONAN_CMAKE_TOOLCHAIN_FILE=["{{ cmake_toolchain_file }}"]
CC={{ cross_compiler_prefix }}-gcc {{ compiler_flags }}
CXX={{ cross_compiler_prefix }}-g++ {{ compiler_flags }}
CPP={{ cross_compiler_prefix }}-gcc -E {{ compiler_flags }}
AS={{ cross_compiler_prefix }}-as
LD={{ cross_compiler_prefix }}-ld {{ compiler_flags }}
GDB={{ cross_compiler_prefix }}-gdb
STRIP={{ cross_compiler_prefix }}-strip
RANLIB={{ cross_compiler_prefix }}-ranlib
OBJCOPY={{ cross_compiler_prefix }}-objcopy
OBJDUMP={{ cross_compiler_prefix }}-objdump
AR={{ cross_compiler_prefix }}-ar
NM={{ cross_compiler_prefix }}-nm
M4=m4
TARGET_PREFIX={{ cross_compiler_prefix }}-
ARCH=arm64
CONFIGURE_FLAGS=--target={{ cross_compiler_prefix }} --host={{ cross_compiler_prefix }} --build=x86_64-linux --with-libtool-sysroot={{ sdk_target_sysroot }}
OE_CMAKE_TOOLCHAIN_FILE={{ os.path.join(yocto_sdk_directory, "sysroots", build_compiler_prefix, "usr", "share", "cmake", "OEToolchainConfig.cmake") }}
OE_CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX="`echo lib | sed -e s/lib//`"
PKG_CONFIG_SYSROOT_MAP={{ sdk_target_sysroot }}
PKG_CONFIG_PATH={{ os.path.join(sdk_target_sysroot, "usr", "lib", "pkgconfig") }}:{{ os.path.join(sdk_target_sysroot, "usr", "share", "pkgconfig") }}
CONFIG_SITE={{ os.path.join(yocto_sdk_directory, "site-config-cortexa72-crypto-poky-linux") }}
OECORE_TARGET_ARCH="aarch64"
OECORE_TARGET_OS="linux"
[buildenv]
PATH=+(path){{ os.path.join(sdk_build_sysroot, "sysroots", build_compiler_prefix, "usr", "bin", cross_compiler_prefix) }}
CC={{ cross_compiler_prefix }}-gcc {{ compiler_flags }}
CXX={{ cross_compiler_prefix }}-g++ {{ compiler_flags }}
CPP={{ cross_compiler_prefix }}-gcc -E {{ compiler_flags }}
AS={{ cross_compiler_prefix }}-as
LD={{ cross_compiler_prefix }}-ld {{ compiler_flags }}
GDB={{ cross_compiler_prefix }}-gdb
STRIP={{ cross_compiler_prefix }}-strip
RANLIB={{ cross_compiler_prefix }}-ranlib
OBJCOPY={{ cross_compiler_prefix }}-objcopy
OBJDUMP={{ cross_compiler_prefix }}-objdump
AR={{ cross_compiler_prefix }}-ar
NM={{ cross_compiler_prefix }}-nm
M4=m4
TARGET_PREFIX={{ cross_compiler_prefix }}-
ARCH=arm64
CONFIGURE_FLAGS=--target={{ cross_compiler_prefix }} --host={{ cross_compiler_prefix }} --build=x86_64-linux --with-libtool-sysroot={{ sdk_target_sysroot }}
OE_CMAKE_TOOLCHAIN_FILE={{ os.path.join(yocto_sdk_directory, "sysroots", build_compiler_prefix, "usr", "share", "cmake", "OEToolchainConfig.cmake") }}
OE_CMAKE_FIND_LIBRARY_CUSTOM_LIB_SUFFIX=`echo lib | sed -e s/lib//`
PKG_CONFIG_SYSROOT_MAP={{ sdk_target_sysroot }}
PKG_CONFIG_PATH={{ os.path.join(sdk_target_sysroot, "usr", "lib", "pkgconfig") }}:{{ os.path.join(sdk_target_sysroot, "usr", "share", "pkgconfig") }}
CONFIG_SITE={{ os.path.join(yocto_sdk_directory, "site-config-cortexa72-crypto-poky-linux") }}
OECORE_TARGET_ARCH=aarch64
OECORE_TARGET_OS=linux
[conf]
tools.build:compiler_executables={"asm": "{{ cross_compiler_prefix }}-as", "c": "{{ cross_compiler_prefix }}-gcc", "cpp": "{{ cross_compiler_prefix }}-g++"}
tools.build:cflags={{ architecture_compiler_flags }}
tools.build:cxxflags={{ architecture_compiler_flags }}
tools.build:exelinkflags={{ architecture_compiler_flags }}
tools.build:sharedlinkflags={{ architecture_compiler_flags }}
tools.build:sysroot={{ sdk_target_sysroot }}
tools.cmake.cmaketoolchain:system_processor=aarch64
tools.cmake.cmaketoolchain:user_toolchain=["{{ cmake_toolchain_file }}"]
jasal82
commented
10 months ago Same issue here. A fix would be appreciated.
Description
I have a Yocto SDK installed on my system which I intend to use for a cross compilation. Relevant information (which is usually sourced via environment-setup-script) is propagated to conan via a profiles tool_requires.
I managed to fix it locally by changing a single line in boost/all/conanfile.py:
Worries are now that this change could lead to a regression, since it basically undoes https://github.com/conan-io/conan-center-index/pull/15154 (issue was https://github.com/conan-io/conan-center-index/issues/15070).
Package and Environment Details
Conan profile
nativecrossSteps to reproduce
yoctoenv/conanfile.py(= updated version of https://github.com/conan-io/conan/issues/5059 + ideas fromandroid-ndk)Logs
original recipe
``` warning: toolset gcc initialization: warning: can not find user-provided command 'aarch64-poky-linux-g++ -mcpu=cortex-a57 -march=armv8-a+crc -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/opt/poky/sysroots/cortexa57-poky-linux' warning: initialized from /workspaces/yocto-sandbox/.conan/data/boost/1.79.0/_/_/source/src/tools/build/user-config.jam:5 /workspaces/yocto-sandbox/.conan/data/b2/4.9.3/_/_/package/4db1be536558d833e52e862fd84d64d75c2b3656/bin/.b2/tools/common.jam:1104: in common.find-compiler from module common error: toolset gcc initialization: error: provided command '"aarch64-poky-linux-g++ -mcpu=cortex-a57 -march=armv8-a+crc -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/opt/poky/sysroots/cortexa57-poky-linux"' not found error: initialized from /workspaces/yocto-sandbox/.conan/data/boost/1.79.0/_/_/source/src/tools/build/user-config.jam:5 ```modified recipe
``` notice: will use 'aarch64-poky-linux-g++' '-mcpu=cortex-a57' '-march=armv8-a+crc' '-fstack-protector-strong' '-O2' '-D_FORTIFY_SOURCE=2' '-Wformat' '-Wformat-security' '-Werror=format-security' '--sysroot=/opt/poky/sysroots/cortexa57-poky-linux' for gcc, conditionfull error log
``` Exporting package recipe boost/1.79.0 exports: File 'conandata.yml' found. Exporting it... boost/1.79.0 exports: Copied 1 '.yml' file: conandata.yml boost/1.79.0: Calling export() boost/1.79.0: Copied 1 '.yml' file: dependencies-1.79.0.yml boost/1.79.0: Calling export_sources() boost/1.79.0: The stored package has not changed boost/1.79.0: Exported revision: 7e3ccedb3882273b9467973efff282d6 Configuration (profile_host): [settings] arch=armv8 build_type=Release compiler=gcc compiler.libcxx=libstdc++11 compiler.version=11.3 os=Linux [options] [build_requires] *: yoctoenv/0.1.0@user/channel [env] Configuration (profile_build): [settings] arch=x86_64 build_type=Release compiler=gcc compiler.libcxx=libstdc++11 compiler.version=11.3 os=Linux [options] [build_requires] [env] boost/1.79.0: Forced build from source boost/1.79.0 (test package): Installing package Requirements boost/1.79.0 from local cache - Cache bzip2/1.0.8 from 'conancenter' - Cache libbacktrace/cci.20210118 from 'conancenter' - Cache zlib/1.2.13 from 'conancenter' - Cache Packages boost/1.79.0:c4b374b9a424b32e9a3de4b28850101d319f40eb - Build bzip2/1.0.8:578ee81484b9a78d79603e4317b520749ee97294 - Cache libbacktrace/cci.20210118:f7bf7f2b9d3c831882a24374763de8125c6889b3 - Cache zlib/1.2.13:f7bf7f2b9d3c831882a24374763de8125c6889b3 - Cache Build requirements b2/4.9.3 from 'conancenter' - Cache yoctoenv/0.1.0@user/channel from local cache - Cache Build requirements packages b2/4.9.3:4db1be536558d833e52e862fd84d64d75c2b3656 - Cache yoctoenv/0.1.0@user/channel:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache Cross-build from 'Linux:x86_64' to 'Linux:armv8' Installing (downloading, building) binaries... b2/4.9.3: Already installed! bzip2/1.0.8: Already installed! libbacktrace/cci.20210118: Already installed! yoctoenv/0.1.0@user/channel: Already installed! zlib/1.2.13: Already installed! boost/1.79.0: Applying build-requirement: yoctoenv/0.1.0@user/channel boost/1.79.0: Applying build-requirement: b2/4.9.3 boost/1.79.0: WARN: Build folder is dirty, removing it: /workspaces/yocto-sandbox/.conan/data/boost/1.79.0/_/_/build/c4b374b9a424b32e9a3de4b28850101d319f40eb boost/1.79.0: Building your package in /workspaces/yocto-sandbox/.conan/data/boost/1.79.0/_/_/build/c4b374b9a424b32e9a3de4b28850101d319f40eb boost/1.79.0: Generator txt created conanbuildinfo.txt boost/1.79.0: Calling generate() boost/1.79.0: Aggregating env generators boost/1.79.0: Calling build() boost/1.79.0: WARN: replace_in_file didn't find pattern '$(>) > $(<)' in '/workspaces/yocto-sandbox/.conan/data/boost/1.79.0/_/_/source/src/libs/stacktrace/build/Jamfile.v2' file. boost/1.79.0: WARN: replace_in_file didn't find pattern '/* thread_local */' in '/workspaces/yocto-sandbox/.conan/data/boost/1.79.0/_/_/source/src/boost/stacktrace/detail/libbacktrace_impls.hpp' file. boost/1.79.0: WARN: replace_in_file didn't find pattern '/* static __thread */' in '/workspaces/yocto-sandbox/.conan/data/boost/1.79.0/_/_/source/src/boost/stacktrace/detail/libbacktrace_impls.hpp' file. boost/1.79.0: WARN: replace_in_file didn't find pattern 'local generic-os = [ set.difference $(all-os) : aix darwin vxworks solaris osf hpux ] ;' in '/workspaces/yocto-sandbox/.conan/data/boost/1.79.0/_/_/source/src/tools/build/src/tools/gcc.jam' file. boost/1.79.0: WARN: replace_in_file didn't find pattern 'local no-threading = android beos haiku sgi darwin vxworks ;' in '/workspaces/yocto-sandbox/.conan/data/boost/1.79.0/_/_/source/src/tools/build/src/tools/gcc.jam' file. boost/1.79.0: WARN: replace_in_file didn't find pattern '