[bug] Cross Compilation not working

powellnorma commented 1 year ago

Environment details

Conan version: 2.0.8

Probably related to https://github.com/conan-io/conan/issues/8652 - However a bit of a different setup, and maybe easier to reproduce.

If anyone knows a quick workaround I'd be thankful too!

Steps to reproduce

conan install --require=libxslt/1.1.34@ --deployer=full_deploy -of=output --build=missing --profile:build android-ndk -o *:shared=True

The android-ndk profile is as described in: https://docs.conan.io/2/examples/cross_build/android/ndk.html

I have however added CC=/path/to/aarch64-linux-android30-clang to [buildenv].

Logs

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux
[options]
*:shared=True

Profile build:
[settings]
arch=armv8
build_type=Release
compiler=clang
compiler.cppstd=14
compiler.libcxx=c++_static
compiler.version=12
os=Android
os.api_level=30
[conf]
tools.android:ndk_path=/opt/android-ndk-r23b
[buildenv]
CC=/opt/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang

======== Computing dependency graph ========
Graph root
    cli
Requirements
    libiconv/1.17#fa54397801cd96911a8294bc5fc76335 - Cache
    libxml2/2.10.3#e49fc911ac79d5b122b4ea3c3a766ae2 - Cache
    libxslt/1.1.34#a324b1b136c75a42bb25008b3b1290a3 - Cache
    zlib/1.3#06023034579559bb64357db3a53f88a4 - Cache
Build requirements
    meson/1.0.0#15586c0ac6f682805875ef903dbe7ee2 - Cache
    ninja/1.11.1#77587f8c8318662ac8e5a7867eb4be21 - Cache
    pkgconf/2.0.3#e1a6e243ef4f1d63f94f8e5bc438ea53 - Cache
Resolved version ranges
    zlib/[>=1.2.11 <2]: zlib/1.3

======== Computing necessary packages ========
Requirements
    libiconv/1.17#fa54397801cd96911a8294bc5fc76335:9a7f5466b6926f6dc790c94d617e893533d5c141#96df27f9eec891238abf2726f7e089c8 - Cache
    libxml2/2.10.3#e49fc911ac79d5b122b4ea3c3a766ae2:ee82171b08dad64dc6df26233389a408acee38cb - Build
    libxslt/1.1.34#a324b1b136c75a42bb25008b3b1290a3:ab9e23777192b0fc94371defb5f76606337b63bc - Build
    zlib/1.3#06023034579559bb64357db3a53f88a4:9a7f5466b6926f6dc790c94d617e893533d5c141#17397451e8779a3c9fd6a23d07923656 - Cache
Build requirements
    meson/1.0.0#15586c0ac6f682805875ef903dbe7ee2:da39a3ee5e6b4b0d3255bfef95601890afd80709#5c8fd51fc33f12e26519674d99afd0e5 - Cache
    ninja/1.11.1#77587f8c8318662ac8e5a7867eb4be21:7f42c4dab96a0a181222694eecf742412e2b5f93#3869de37d5f26f32d8120ffe64c0c1e7 - Cache
    pkgconf/2.0.3#e1a6e243ef4f1d63f94f8e5bc438ea53:17051ede4c9d5563872b017b78fd64c6ea3b51d6 - Build

======== Installing packages ========
libiconv/1.17: Already installed! (1 of 7)
ninja/1.11.1: Already installed! (2 of 7)
zlib/1.3: Already installed! (3 of 7)
meson/1.0.0: Already installed! (4 of 7)

-------- Installing package pkgconf/2.0.3 (5 of 7) --------
pkgconf/2.0.3: Building from source
pkgconf/2.0.3: Package pkgconf/2.0.3:17051ede4c9d5563872b017b78fd64c6ea3b51d6
pkgconf/2.0.3: Copying sources to build folder
pkgconf/2.0.3: Building your package in /home/user/.conan2/p/b/pkgcofe31e818c29f1/b
pkgconf/2.0.3: Calling generate()
pkgconf/2.0.3: Generators folder: /home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release/conan
pkgconf/2.0.3: Generating aggregated env files
pkgconf/2.0.3: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
pkgconf/2.0.3: Calling build()
pkgconf/2.0.3: Apply patch (file): patches/1.9.3-0003-PKG_CONF_PATH-allow-colon+semicolon-separator.patch
pkgconf/2.0.3: Meson configure cmd: meson setup --native-file "/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release/conan/conan_meson_native.ini" "/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release" "/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/src" -Dprefix="/home/user/.conan2/p/b/pkgcofe31e818c29f1/p"
pkgconf/2.0.3: RUN: meson setup --native-file "/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release/conan/conan_meson_native.ini" "/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release" "/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/src" -Dprefix="/home/user/.conan2/p/b/pkgcofe31e818c29f1/p"
The Meson build system
Version: 1.0.0
Source dir: /home/user/.conan2/p/b/pkgcofe31e818c29f1/b/src
Build dir: /home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release
Build type: native build
../src/meson.build:1: WARNING: Keyword argument "default_options" defined multiple times.
WARNING: This will be an error in future Meson releases.
Project name: pkgconf
Project version: 2.0.3

../src/meson.build:1:0: ERROR: Could not invoke sanity test executable: [Errno 8] Exec format error: '/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release/meson-private/sanitycheckc.exe'.

A full log can be found at /home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release/meson-logs/meson-log.txt

pkgconf/2.0.3: ERROR: 
Package '17051ede4c9d5563872b017b78fd64c6ea3b51d6' build failed
pkgconf/2.0.3: WARN: Build folder /home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release
*********************************************************
Recipe 'pkgconf/2.0.3' cannot build its binary
It is possible that this recipe is not Conan 2.0 ready
If the recipe comes from ConanCenter check: https://conan.io/cci-v2.html
If it is your recipe, check if it is updated to 2.0
*********************************************************

ERROR: pkgconf/2.0.3: Error in build() method, line 89
    meson.configure()
    ConanException: Error 1 while executing

memsharded commented 1 year ago

Hi @powellnorma

Thanks for your report.

Could you please try to cross-compile with the same setup a meson template with conan new meson_lib -d name=pkg -d version=0.1 and also conan new meson_exe -d name=app -d version=0.1? That might help to reduce the problem a little bit, maybe it is specific to the ConanCenter recipe. Depending on your setup, you might need to add the build_requirements() to meson, ninja and pkgconf in the recipe, to inject them, if you don't have them in the system.

memsharded commented 1 year ago

Oh, sorry, I think there is some misconfiguration, your build profile is wrong. The build profile is for your current build machine. It seems that you have swapped the host and build profiles

powellnorma commented 1 year ago

Thank you for your answer.

Hm, I have tried this before: conan install --require=libxslt/1.1.34@ --deployer=full_deploy -of=output --build=missing --profile:host my-ndk -o *:shared=True But while it works, the resulting .so is for x86_64:

readelf -h output/full_deploy/host/libxslt/1.1.34/Release/armv8/lib/libxslt.so 
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          287248 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         11
  Size of section headers:           64 (bytes)
  Number of section headers:         31
  Section header string table index: 30

What am I missing here?

franramirez688 commented 1 year ago

Hi @powellnorma ,

Could you please print your my-ndk profile? Even if you print the whole Conan log, it would be really helpful to see what's happening.

powellnorma commented 1 year ago

Even if you print the whole Conan log

I thought I have done so, or am I missing something? And I think the log contains the my-ndk profile (there its the Profile build). Have you tried reproducing this with the profiles given in the log, and the command I mentioned? Thank you for your help.

franramirez688 commented 1 year ago

Hi @powellnorma

Then I thought I was missing something because I felt you tried another thing. The problem is, like @memsharded said, you swapped the host/build profiles. If you try this, it should work:

$ conan install --require=libxslt/1.1.34@ --deployer=full_deploy -of=output --build=missing --profile:host android-ndk -o *:shared=True

# .....

pkgconf/2.0.3: RUN: meson setup --cross-file "/root/.conan2/p/b/pkgco9f440b9428391/b/build-release/conan/conan_meson_cross.ini" "/root/.conan2/p/b/pkgco9f440b9428391/b/build-release" "/root/.conan2/p/b/pkgco9f440b9428391/b/src" -Dprefix="/root/.conan2/p/b/pkgco9f440b9428391/p" --reconfigure
The Meson build system
Version: 1.0.0
Source dir: /root/.conan2/p/b/pkgco9f440b9428391/b/src
Build dir: /root/.conan2/p/b/pkgco9f440b9428391/b/build-release
Build type: cross build
../src/meson.build:1: WARNING: Keyword argument "default_options" defined multiple times.
WARNING: This will be an error in future Meson releases.
Project name: pkgconf
Project version: 2.0.3
C compiler for the host machine: /home/develop/infra_scripts/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang (clang 17.0.2 "Android (10087095, +pgo, +bolt, +lto, -mlgo, based on r487747c) clang version 17.0.2 (https://android.googlesource.com/toolchain/llvm-project d9f89f4d16663d5012e5c09495f3b30ece3d2362)")
C linker for the host machine: /home/develop/infra_scripts/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang ld.lld 17.0.2
C compiler for the build machine: cc (gcc 11.1.0 "cc (Ubuntu 11.1.0-1ubuntu1~20.04) 11.1.0")
C linker for the build machine: cc ld.bfd 2.34

#...

pkgconf/2.0.3: package(): Packaged 2 files: COPYING, pkgconf
pkgconf/2.0.3: package(): Packaged 1 '.m4' file: pkg.m4
pkgconf/2.0.3: Created package revision 1f0647bbdc85ea66ee94859e281f128d
pkgconf/2.0.3: Package 'c0b621fd4b3199fe05075171573398833dba85f4' created
pkgconf/2.0.3: Full package reference: pkgconf/2.0.3#776d4ee29b7e98f76958a3c0e6f8cc86:c0b621fd4b3199fe05075171573398833dba85f4#1f0647bbdc85ea66ee94859e281f128d
pkgconf/2.0.3: Package folder /root/.conan2/p/b/pkgco389b07fd865a9/p
pkgconf/2.0.3: Appending PATH env var: /root/.conan2/p/b/pkgco389b07fd865a9/p/bin
pkgconf/2.0.3: Setting PKG_CONFIG env var: /root/.conan2/p/b/pkgco389b07fd865a9/p/bin/pkgconf
pkgconf/2.0.3: WARN: The use of 'unix_path_legacy_compat' is deprecated in Conan 2.0 and does not perform path conversions. This is retained for compatibility with Conan 1.x and will be removed in a future version.
pkgconf/2.0.3: Appending AUTOMAKE_CONAN_INCLUDES env var: /root/.conan2/p/b/pkgco389b07fd865a9/p/bin/aclocal
WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: deprecated:     'env_info' used in: pkgconf/2.0.3, autoconf/2.71, automake/1.16.5, m4/1.4.19
WARN: deprecated:     'user_info' used in: automake/1.16.5

Let me know if you find another problem or have any other doubts.

powellnorma commented 1 year ago

My problem is that the command you use results in x86_64 binaries, not arm64 binaries (see here).

What is your output of readelf -h output/full_deploy/host/libxslt/1.1.34/Release/armv8/lib/libxslt.so ?

I otherwise use exactly the same profile my-ndk as in the issue description, but with host and build profiles swapped.

franramirez688 commented 1 year ago

Okay, let's simplify the example to see if it could be a problem with the recipe.

My ndk_profile is this one:

ndk_profile

[settings]
arch=armv8
build_type=Release
compiler=clang
compiler.cppstd=14
compiler.libcxx=c++_static
compiler.version=12
os=Android
os.api_level=30
[conf]
tools.android:ndk_path=/home/develop/infra_scripts/android-ndk-r26
[buildenv]
CC=/home/develop/infra_scripts/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang
CXX=/home/develop/infra_scripts/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang++

Then, let's start with the very basic "Hello world" example:

$ conan new cmake_lib -d name=hello -d version=1.0
$ conan create . -pr:h ndk_profile -o *:shared=True
======== Input profiles ========
Profile host:
[settings]
arch=armv8
build_type=Release
compiler=clang
compiler.cppstd=14
compiler.libcxx=c++_static
compiler.version=12
os=Android
os.api_level=30
[options]
*:shared=True
[conf]
tools.android:ndk_path=/home/develop/infra_scripts/android-ndk-r26
[buildenv]
CC=/home/develop/infra_scripts/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang
CXX=/home/develop/infra_scripts/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang++

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux

# ....
Scanning dependencies of target example
[ 50%] Building CXX object CMakeFiles/example.dir/src/example.cpp.o
[100%] Linking CXX executable example
[100%] Built target example

======== Testing the package: Executing test ========
hello/1.0 (test package): Running test()

Now, let's check the arch for those binaries:

$ readelf -h /root/.conan2/p/b/helloe256013990b03/p/lib/libhello.so
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          1414112 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         10
  Size of section headers:           64 (bytes)
  Number of section headers:         34
  Section header string table index: 32

$ readelf -h test_package/build/clang-12-armv8-14-release/example
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x3a58
  Start of program headers:          64 (bytes into file)
  Start of section headers:          160512 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         12
  Size of section headers:           64 (bytes)
  Number of section headers:         37
  Section header string table index: 35

As you can observe, both binaries are AArch64, so that's what we want. Could you try those steps to see if it's working on your side?

UPDATED: By the record, if you want to try another simple example with Meson too, the steps would be mostly the same:

$ conan new meson_lib -d name=hello -d version=1.0
$ PKG_CONFIG=/usr/bin/pkg-config conan create . -pr:h ndk_profile -o *:shared=True

powellnorma commented 1 year ago

Yes, I get AArch64 too for your first example. I have one question unrelated to this problem: When I do conan install . -pr:h my-ndk --deployer=full_deploy -of=output -o *:shared=True, why is libhello.so not put into the output directory?

The second example fails, because meson is not found:

-------- Installing package hello/1.0 (1 of 1) --------
hello/1.0: Building from source
hello/1.0: Package hello/1.0:1bd62d20ef01783ab4ee0db2193cdf5391f77e46
hello/1.0: Copying sources to build folder
hello/1.0: Building your package in /home/user/.conan2/p/b/hello190a5dba72cf0/b
hello/1.0: Calling generate()
hello/1.0: Generators folder: /home/user/.conan2/p/b/hello190a5dba72cf0/b/build-release/conan
hello/1.0: Generating aggregated env files
hello/1.0: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
hello/1.0: Calling build()
hello/1.0: Meson configure cmd: meson setup --cross-file "/home/user/.conan2/p/b/hello190a5dba72cf0/b/build-release/conan/conan_meson_cross.ini" "/home/user/.conan2/p/b/hello190a5dba72cf0/b/build-release" "/home/user/.conan2/p/b/hello190a5dba72cf0/b" -Dprefix="/home/user/.conan2/p/b/hello190a5dba72cf0/p"
hello/1.0: RUN: meson setup --cross-file "/home/user/.conan2/p/b/hello190a5dba72cf0/b/build-release/conan/conan_meson_cross.ini" "/home/user/.conan2/p/b/hello190a5dba72cf0/b/build-release" "/home/user/.conan2/p/b/hello190a5dba72cf0/b" -Dprefix="/home/user/.conan2/p/b/hello190a5dba72cf0/p"
/bin/sh: line 1: meson: command not found

On your setup, is the output/full_deploy/host/libxslt/1.1.34/Release/armv8/lib/libxslt.so generated as AArch64, or also as Advanced Micro Devices X86-64?

franramirez688 commented 1 year ago

Yes, I get AArch64 too for your first example.

Then, the cross-compilation seems to be working on your side but not for the libxslt recipe, so it seems to be a problem with the recipe, doesn't it?

I have one question unrelated to this problem: When I do conan install . -pr:h my-ndk --deployer=full_deploy -of=output -o *:shared=True, why is libhello.so not put into the output directory?

Because Conan is building and saving all the libraries in the Conan cache.

Let me bring this piece of documentation about the conan install:

This command does the following:

* Compute the whole dependency graph, for the current configuration defined by settings, options, profiles and configuration. It resolves version ranges, transitive dependencies, conditional requirements, etc, to build the dependency graph.
* Evaluate the existence of binaries for every package in the graph, whether or not there are precompiled binaries to download, or if they should be built from sources (as directed by the --build argument). If binaries are missing, it will not recompute the dependency graph to try to fallback to previous versions that contain binaries for that configuration. If a certain dependency version is desired, it should be explicitly required.
* Download precompiled binaries, or build binaries from sources in the local cache, in the right order for the dependency graph.
* Create the necessary files as requested by the “generators”, so build systems and other tools can locate the locally installed dependencies
* Optionally, execute the desired deployers.

I recommend you to have a look at these docs links:

The second example fails, because meson is not found:

Yeah, you need to have installed Meson on your system (for instance, pip install meson) or require meson/x.x.x in the build_requirements function as a tool_requires("meson/x.x.x").

On your setup, is the output/full_deploy/host/libxslt/1.1.34/Release/armv8/lib/libxslt.so generated as AArch64, or also as Advanced Micro Devices X86-64?

I'm getting errors compiling the whole dependency graph. My logs above were only meant to show how pkgconf was correctly built as your logs shown in the issue were these ones:

pkgconf/2.0.3: RUN: meson setup --native-file "/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release/conan/conan_meson_native.ini" "/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release" "/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/src" -Dprefix="/home/user/.conan2/p/b/pkgcofe31e818c29f1/p"
The Meson build system
Version: 1.0.0
Source dir: /home/user/.conan2/p/b/pkgcofe31e818c29f1/b/src
Build dir: /home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release
Build type: native build
../src/meson.build:1: WARNING: Keyword argument "default_options" defined multiple times.
WARNING: This will be an error in future Meson releases.
Project name: pkgconf
Project version: 2.0.3

../src/meson.build:1:0: ERROR: Could not invoke sanity test executable: [Errno 8] Exec format error: '/home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release/meson-private/sanitycheckc.exe'.

A full log can be found at /home/user/.conan2/p/b/pkgcofe31e818c29f1/b/build-release/meson-logs/meson-log.txt

My error running the command:

$ conan build --require=libxslt/1.1.34@ --deployer=full_deploy -of=output --build=missing -pr:h ../ndk_profile -o *:shared=True

# ...

[ 87%] Building C object CMakeFiles/zlib.dir/zutil.c.o
[ 90%] Building C object CMakeFiles/zlibstatic.dir/uncompr.c.o
[ 84%] Building C object CMakeFiles/zlibstatic.dir/trees.c.o
[ 93%] Building C object CMakeFiles/zlibstatic.dir/zutil.c.o
[ 96%] Linking C shared library libz.so
ld.lld: error: version script assignment of 'local' to symbol 'gz_intmax' failed: symbol not defined
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/zlib.dir/build.make:313: libz.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:124: CMakeFiles/zlib.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[100%] Linking C static library libzlibstatic.a
[100%] Built target zlibstatic
make: *** [Makefile:160: all] Error 2

Anyway, it does not matter. My doubt is if you pointed out this log https://github.com/conan-io/conan/issues/14796#issuecomment-1731203474 then you should have some "successful" ones, could you please print them here?

powellnorma commented 1 year ago

Because Conan is building and saving all the libraries in the Conan cache.

Yes, but I thought --deployer=full_deploy -of=output should copy everything to that directory? It did that in the libxslt case.

My doubt is if you pointed out this log https://github.com/conan-io/conan/issues/14796#issuecomment-1731203474 then you should have some "successful" ones, could you please print them here?

Here is the log of the "successful" build but with wrong resulting architecture (Advanced Micro Devices X86-64 instead of AArch64):

Log

``` conan install --require=libxslt/1.1.34@ --deployer=full_deploy -of=output --build=missing --profile:host my-ndk -o *:shared=True ======== Input profiles ======== Profile host: [settings] arch=armv8 build_type=Release compiler=clang compiler.cppstd=14 compiler.libcxx=c++_static compiler.version=12 os=Android os.api_level=30 [options] *:shared=True [conf] tools.android:ndk_path=/opt/android-ndk-r23b [buildenv] CC=/opt/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang Profile build: [settings] arch=x86_64 build_type=Release compiler=gcc compiler.cppstd=gnu17 compiler.libcxx=libstdc++11 compiler.version=12 os=Linux ======== Computing dependency graph ======== Graph root cli Requirements libiconv/1.17#fa54397801cd96911a8294bc5fc76335 - Cache libxml2/2.10.3#e49fc911ac79d5b122b4ea3c3a766ae2 - Cache libxslt/1.1.34#a324b1b136c75a42bb25008b3b1290a3 - Cache zlib/1.3#06023034579559bb64357db3a53f88a4 - Cache Build requirements meson/1.0.0#15586c0ac6f682805875ef903dbe7ee2 - Cache ninja/1.11.1#77587f8c8318662ac8e5a7867eb4be21 - Cache pkgconf/2.0.3#e1a6e243ef4f1d63f94f8e5bc438ea53 - Cache Resolved version ranges zlib/[>=1.2.11 <2]: zlib/1.3 ======== Computing necessary packages ======== Requirements libiconv/1.17#fa54397801cd96911a8294bc5fc76335:4d6c4d71faaded7558933acaef6e31b58f585e77#c1416c85f281d0175deedbe0a9db8057 - Cache libxml2/2.10.3#e49fc911ac79d5b122b4ea3c3a766ae2:fad5d6caefc2d69ecbbd310666fc3d5ce5e22203#6b36147d9ac93c78fdbbabba8bc4ffcb - Cache libxslt/1.1.34#a324b1b136c75a42bb25008b3b1290a3:47cf4f9ee19143c3c5a9c284ec77f260dd84211d#b1dbbb89fb44415956638f0ac89914b9 - Cache zlib/1.3#06023034579559bb64357db3a53f88a4:4d6c4d71faaded7558933acaef6e31b58f585e77#54e935bd24bc743129a23bbd485f2dec - Cache Build requirements Skipped binaries meson/1.0.0, ninja/1.11.1, pkgconf/2.0.3 ======== Installing packages ======== libiconv/1.17: Already installed! (1 of 4) zlib/1.3: Already installed! (2 of 4) libxml2/2.10.3: Already installed! (3 of 4) libxml2/2.10.3: Appending PATH environment variable: /home/user/.conan2/p/b/libxm7b1356aa20246/p/bin libxslt/1.1.34: Already installed! (4 of 4) WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: WARN: deprecated: 'cpp_info.names' used in: zlib/1.3, libxslt/1.1.34, libiconv/1.17, libxml2/2.10.3 WARN: deprecated: 'env_info' used in: libxslt/1.1.34, libiconv/1.17, libxml2/2.10.3 WARN: deprecated: 'cpp_info.filenames' used in: libxml2/2.10.3 WARN: deprecated: 'cpp_info.build_modules' used in: libxml2/2.10.3 ======== Finalizing install (deploy, generators) ======== cli: Conan built-in full deployer to /tmp/c4/output cli: Generating aggregated env files cli: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh'] Install finished successfully ```

franramirez688 commented 1 year ago

Yes, but I thought --deployer=full_deploy -of=output should copy everything to that directory? It did that in the libxslt case.

Oh, yeah, sorry, I didn't get what you said. So, basically, there is a misunderstanding about those parameters:

  -of OUTPUT_FOLDER, --output-folder OUTPUT_FOLDER
                        The root output folder for generated and build files
  -d DEPLOYER, --deployer DEPLOYER
                        Deploy using the provided deployer to the output folder
  --deployer-folder DEPLOYER_FOLDER
                        Deployer output folder, base build folder by default if not set

Then, -of is used by Conan to put all its own generated files there, but not for the deployed ones. For that, you have to use the --deployer-folder YOUR_FOLDER if you want to change the default full_deploy folder.

Here is the log of the "successful" build but with wrong resulting architecture (Advanced Micro Devices X86-64 instead of AArch64):

If everything is OK, then it has to be a problem with the recipe. Have you checked the rest of the libraries? zlib, libiconv, etc.? Are they built for AArch64?

UPDATE: I pressed the wrong button, sorry about closing the issue a few seconds ago 😅

powellnorma commented 1 year ago

Then, -of is used by Conan to put all its own generated files there, but not for the deployed ones. For that, you have to use the --deployer-folder YOUR_FOLDER if you want to change the default full_deploy folder.

Ok but I think base build folder by default if not set is root output folder for generated and build files, so setting -of would also set the deployer-folder?

I tried --deployer-folder output-deploy and that folder got created but was empty - Maybe the recipe does not declare libhello.so as output somehow?

Have you checked the rest of the libraries? zlib, libiconv, etc.? Are they built for AArch64?

Only libz.so has the correct architecture:

File: ./output/full_deploy/host/libiconv/1.17/Release/armv8/lib/libcharset.so
  Machine:                           Advanced Micro Devices X86-64
File: ./output/full_deploy/host/libiconv/1.17/Release/armv8/lib/libiconv.so
  Machine:                           Advanced Micro Devices X86-64
File: ./output/full_deploy/host/zlib/1.3/Release/armv8/lib/libz.so
  Machine:                           AArch64
File: ./output/full_deploy/host/libxml2/2.10.3/Release/armv8/lib/libxml2.so
  Machine:                           Advanced Micro Devices X86-64
File: ./output/full_deploy/host/libxslt/1.1.34/Release/armv8/lib/libxslt.so
  Machine:                           Advanced Micro Devices X86-64
File: ./output/full_deploy/host/libxslt/1.1.34/Release/armv8/lib/libexslt.so
  Machine:                           Advanced Micro Devices X86-64

memsharded commented 1 year ago

@franramirez688 this could be related to the recipes:

Some issue with the recipes themselves
Some issue with build system integrations used in recipes, other than CMake (worth checking each one of the above packages what integration they are using)

franramirez688 commented 1 year ago

@powellnorma sorry for the delay in replying to your comments.

I tried --deployer-folder output-deploy and that folder got created but was empty - Maybe the recipe does not declare libhello.so as output somehow?

Have you tried the latest Conan 2.0.x? It's only to be sure that it could be a bug already solved by newer versions.

Only libz.so has the correct architecture:

Okay, it could be related to the recipes themselves and their build system integrations. I'll try to have a quick look before transferring to CCI repository.

powellnorma commented 1 year ago

No worries

$ conan --version
Conan version 2.0.8

memsharded commented 1 year ago

Could you please update to the latest 2.0.13 and retry? Only the latest patch of each minor version is recommended for usage (see https://docs.conan.io/2/introduction.html#stable: Only the latest released patch (major.minor.patch) of every minor version is supported and stable.)

powellnorma commented 1 year ago

Ok I now have tried it with 2.0.13, but no change in behavior.

memsharded commented 1 year ago

I am trying to replicate (the --deployer-folder part, the build-system integrations check for the architecture @franramirez688 is having a look), and it seems to work fine:

$ conan install --requires="cmake/[>3]" --deployer=full_deploy --deployer-folder=mydeploy
$ mydeploy\full_deploy\host\cmake\3.27.7\x86_64\bin\cmake --version
cmake version 3.27.7

CMake suite maintained and supported by Kitware (kitware.com/cmake).

As you can see, the mydeploy folder is not empty, it will contain the cmake executables. Can you please double check about the deployer folder and let me know? Thanks!

powellnorma commented 1 year ago

Yes, the command you used correctly "fills" mydeploy. I was just wondering why for

conan new cmake_lib -d name=hello -d version=1.0
conan install . -pr:h my-ndk --deployer=full_deploy -of=output -o *:shared=True
conan install . -pr:h my-ndk --deployer=full_deploy --deployer-folder=mydeploy -o *:shared=True

The libhello.so is not placed in neither output nor mydeploy. I guess in the recipe libhello.so is somehow not marked for "deployment"?

memsharded commented 1 year ago

I see what you mean. The hello/1.0 is not a package at all that can be deployed. In your case, it is a consumer application (it doesn't even need the build() method or the package() method to do the conan install). This consumer application is installing its dependencies, and its dependencies are the thing that are deployed. The libhello.so doesn't even exist until you conan install the dependencies, and then execute cmake ... to build it! Then, it is impossible that conan install --deployer... will deploy it, because libhello.so doesn't even exist at this point.

I hope this clarifies the issue.

What is possible is to conan create the hello/0.1 package, then conan install --requires=hello/0.1 --deployer=..., because that creates a full package, and then that package is installed and deployed (but the package has been created, and it will already contain the libhello.so already built.

powellnorma commented 1 year ago

Okay, so "consumer application" basically means the "thing that consumes conan packages", and the idea is that this "consumer application" gets "manually" build by the user, NOT conan itself? Conan only ever builds and "provides" dependencies, never the "final" package, right? So if we want to use conan solely to build library X (for which a conan recipe already exists), we have to "trick" conan by acting as if that library X is "just a dependency", even though its actually the final "target"?

And "conan install ." is similar to "npm install", in that it installs the "dependencies" to "run" (in the case of conan "build") the app?

What is possible is to conan create the hello/0.1 package, then conan install --requires=hello/0.1 --deployer=...

So something like this:

conan new cmake_lib -d name=hello -d version=1.0
conan export .
conan install --require=hello/1.0 --deployer=full_deploy -of=output --build=missing -pr:h my-ndk -o *:shared=True

What just found out, the following works too:

conan new cmake_lib -d name=hello -d version=1.0
conan build . --deployer=full_deploy -of=output --build=missing -pr:h my-ndk -o *:shared=True

I think it would be "usefull"/intuitive to be able to do something like this too:

conan build libxslt/1.1.34@ --deployer=full_deploy -of=output --build=missing --profile:build android-ndk -o *:shared=True

That way one would make it "explicit" that one wants to just build that (existing) package. But it fails with "path not found". I guess this is not possible? (But its not a big problem, since install --requires=libxslt/1.1.34@ works)

Thank you for your explanation, I think my understanding is a bit clearer now :)

memsharded commented 1 year ago

I think you understood it now better.

We actually had the syntax conan install name/version@ and conan build . name/version@ in Conan 1.0. And we decided to change it in Conan 2.0 to make it explicit and did it as conan install --requires=name/version and conan build . --name=name --version=version:

conan install --requires=name/version --build=missing: Install the name/version package and its dependencies
conan build . --name=name --version=version: build the current consumer application (not a package), installing first its dependencies. It will call the conanfile.py that is found in the current dir build() method, but not the package() method.

The CLI in 2.0 is not completely unambiguous, and it avoids a lot of users errors that were very confusing like "I cannot find a conanfile.py in the folder /name/version".

powellnorma commented 1 year ago

What I had in mind, was to be able to use conan build <pkg> as a way to build a remote <pkg> (not downloaded) just as conan install --requires=<pkg> does it.

But I am not sure if that would be a good idea, maybe it would add more confusion.

Is it possible to use conan similarly to e.g. npm install -g <pkg> or cargo install <pkg>, in that the resulting binary gets added/symlinked to a directory that can be added to $PATH? If not are there plans to add this?

memsharded commented 1 year ago

Is it possible to use conan similarly to e.g. npm install -g or cargo install , in that the resulting binary gets added/symlinked to a directory that can be added to $PATH? If not are there plans to add this?

I am afraid not. conan install can operate over conanfile.txt that doesn't even have a build() method, or over conanfile.py without build() method too. The issue is that in C++ the dependency/package manager (Conan) is one thing, and the build system (CMake, MSBuild, Make, etc) is another, separate thing, while cargo is both the build system and dependency manager.

So conan install will keep just installing dependencies, but nothing more, not building the local project. The local project can be built:

Locally with conan build
In the cache with conan create

We could discuss if we wanted the conan build command we want it to do more. But as Conan doesn't have the model of what is being built, because that is just known exclusively by the build system, there is little to do automatically. Until the package is created with package() and package_info() is called, there is little information. And those methods are what are called with conan create (packages need to be somewhere, that is the Conan cache build that happens with conan create)

If we talk about conan install --requires=pkg/name, then it is enough to do a -g VirtualRunEnv to have a virtualenv file with the PATH env var.

powellnorma commented 1 year ago

If we talk about conan install --requires=pkg/name, then it is enough to do a -g VirtualRunEnv to have a virtualenv file with the PATH env var.

Yes, basically if there is a program already on canon, in some scenarios it could be nice to be able to "just" install/compile it. Currently it feels to me as if this functionality is a bit hidden behind options/flags new users don't know about.

OTOH, canon seems to be more of a dependency manager (for a project) than a package manager by itself (for the system).

the build-system integrations check for the architecture @franramirez688 is having a look

Okay, I will keep the issue open then. IIUC it is a bug in the recipe?

franramirez688 commented 1 year ago

Hi @powellnorma

I just tried to install libiconv as I was having problems with zlib cross-compilation, and this is the result:

$ conan install --require=libiconv/1.17 --deployer=full_deploy -of=output --build=missing -pr:h ndk-profile -o *:shared=True
Profile host:
[settings]
arch=armv8
build_type=Release
compiler=clang
compiler.cppstd=14
compiler.libcxx=c++_static
compiler.version=12
os=Android
os.api_level=30
[options]
*:shared=True
[conf]
tools.android:ndk_path=/home/develop/infra_scripts/android-ndk-r26
[buildenv]
CC=/home/develop/infra_scripts/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux

.........
$ readelf -h output/full_deploy/host/libiconv/1.17/Release/armv8/lib/libcharset.so
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          4208 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         22
  Section header string table index: 20

$ readelf -h output/full_deploy/host/libiconv/1.17/Release/armv8/lib/libiconv.so
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          974544 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         10
  Size of section headers:           64 (bytes)
  Number of section headers:         24
  Section header string table index: 22

To ensure that it's a fresh install, please remove libiconv before:

$ conan remove libiconv

Let me know the whole output log (you can attach a txt file if the log is huge).

powellnorma commented 1 year ago

Interesting, now I get this error: ld: error: --fix-cortex-a53-843419 is only supported on AArch64 targets

$ conan install --require=libiconv/1.17 --deployer=full_deploy -of=output --build=missing -pr:h my-ndk -o *:shared=True
======== Input profiles ========
Profile host:
[settings]
arch=armv8
build_type=Release
compiler=clang
compiler.cppstd=14
compiler.libcxx=c++_static
compiler.version=12
os=Android
os.api_level=30
[options]
*:shared=True
[conf]
tools.android:ndk_path=/opt/android-ndk-r23b
[buildenv]
CC=/opt/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang
CXX=/opt/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang++

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux

======== Computing dependency graph ========
Graph root
    cli
Requirements
    libiconv/1.17#fa54397801cd96911a8294bc5fc76335 - Cache

======== Computing necessary packages ========
Requirements
    libiconv/1.17#fa54397801cd96911a8294bc5fc76335:4d6c4d71faaded7558933acaef6e31b58f585e77 - Build

======== Installing packages ========

-------- Installing package libiconv/1.17 (1 of 1) --------
libiconv/1.17: Building from source
libiconv/1.17: Package libiconv/1.17:4d6c4d71faaded7558933acaef6e31b58f585e77
libiconv/1.17: Copying sources to build folder
libiconv/1.17: Building your package in /home/user/.conan2/p/b/libiccc21e3a31ee85/b
libiconv/1.17: Calling generate()
libiconv/1.17: Generators folder: /home/user/.conan2/p/b/libiccc21e3a31ee85/b/build-release/conan
libiconv/1.17: Generating aggregated env files
libiconv/1.17: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
libiconv/1.17: Calling build()
libiconv/1.17: apply_conandata_patches(): No patches defined in conandata
libiconv/1.17: Calling:
 > "/home/user/.conan2/p/b/libiccc21e3a31ee85/b/src/configure" --enable-shared --disable-static --prefix=/ '--bindir=${prefix}/bin' '--sbindir=${prefix}/bin' '--libdir=${prefix}/lib' '--includedir=${prefix}/include' '--oldincludedir=${prefix}/include' --host=aarch64-linux-android --build=x86_64-linux-gnu 
libiconv/1.17: RUN: "/home/user/.conan2/p/b/libiccc21e3a31ee85/b/src/configure" --enable-shared --disable-static --prefix=/ '--bindir=${prefix}/bin' '--sbindir=${prefix}/bin' '--libdir=${prefix}/lib' '--includedir=${prefix}/include' '--oldincludedir=${prefix}/include' --host=aarch64-linux-android --build=x86_64-linux-gnu 
checking for a BSD-compatible install... /usr/bin/install -c

[..]

/bin/sh ../libtool --mode=link /opt/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang  -O3 iconv_no_i18n.o ../srclib/libicrt.a ../lib/libiconv.la  -o iconv_no_i18n
libtool: link: /opt/android-ndk-r23b/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang -O3 iconv_no_i18n.o -o .libs/iconv_no_i18n  ../srclib/libicrt.a ../lib/.libs/libiconv.so -L//lib
ld: error: --fix-cortex-a53-843419 is only supported on AArch64 targets
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:78: iconv_no_i18n] Error 1
make[1]: Leaving directory '/home/user/.conan2/p/b/libicdb6ad54a2e9dd/b/build-release/src'
make: *** [Makefile:35: all] Error 2

libiconv/1.17: ERROR: 
Package '4d6c4d71faaded7558933acaef6e31b58f585e77' build failed
libiconv/1.17: WARN: Build folder /home/user/.conan2/p/b/libicdb6ad54a2e9dd/b/build-release
*********************************************************
Recipe 'libiconv/1.17' cannot build its binary
It is possible that this recipe is not Conan 2.0 ready
If the recipe comes from ConanCenter, report it at https://github.com/conan-io/conan-center-index/issues
If it is your recipe, check if it is updated to 2.0
*********************************************************

ERROR: libiconv/1.17: Error in build() method, line 126
    autotools.make()
    ConanException: Error 2 while executing

powellnorma commented 1 year ago

Let me know the whole output log

Where do I find that log?

franramirez688 commented 1 year ago

ld: error: --fix-cortex-a53-843419 is only supported on AArch64 targets

Okay, it could be related to the NDK version. Could you please update your NDK to the r26 one? Only to be sure that it's something isolated and that it's fixed in newer NDK versions.

Where do I find that log?

Yeah, sorry, I meant exactly the Conan output that you already copied and pasted in your comment above https://github.com/conan-io/conan/issues/14796#issuecomment-1761374230 Thanks!

powellnorma commented 1 year ago

Same error with ndk 26:

$ conan install --require=libiconv/1.17 --deployer=full_deploy -of=output --build=missing -pr:h my-ndk-26 -o *:shared=True
[..]
libtool: link: /opt/android-ndk-r26/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android30-clang -O3 iconv_no_i18n.o -o .libs/iconv_no_i18n  ../srclib/libicrt.a ../lib/.libs/libiconv.so -L//lib
ld.lld: error: --fix-cortex-a53-843419 is only supported on AArch64 targets
clang-17: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [Makefile:78: iconv_no_i18n] Error 1
make[1]: Leaving directory '/home/user/.conan2/p/b/libic3d80ddcd8a26e/b/build-release/src'
make: *** [Makefile:35: all] Error 2

libiconv/1.17: ERROR: 
Package '4d6c4d71faaded7558933acaef6e31b58f585e77' build failed
libiconv/1.17: WARN: Build folder /home/user/.conan2/p/b/libic3d80ddcd8a26e/b/build-release
*********************************************************
Recipe 'libiconv/1.17' cannot build its binary
It is possible that this recipe is not Conan 2.0 ready
If the recipe comes from ConanCenter, report it at https://github.com/conan-io/conan-center-index/issues
If it is your recipe, check if it is updated to 2.0
*********************************************************

ERROR: libiconv/1.17: Error in build() method, line 126
    autotools.make()
    ConanException: Error 2 while executing

franramirez688 commented 1 year ago

@powellnorma If you try the previous command instead with --require=libxml2/2.11.5, is the result the same?

powellnorma commented 1 year ago

Yes, I get the ld.lld: error: --fix-cortex-a53-843419 is only supported on AArch64 targets error there too

franramirez688 commented 1 year ago

@powellnorma I'm not sure if that could be something related to some missing configuration in your system. Anyway, regarding this issue, I think that we could close it as it's not a problem for the Conan client.

Have you searched for similar issues with that linker error?

powellnorma commented 1 year ago

What I don't understand.. Why did it first build without errors but produced x86 binaries, and now it fails?

Are you able to run the commands on your system fine without problems? Maybe we should try to use a Docker environment and test the commands there (to rule out problems due to system config)?

franramirez688 commented 1 year ago

What I don't understand... Why did it first build without errors but produced x86 binaries, and now it fails?

I'm not sure. Perhaps, if you start cleaning up your cache, you installed a fresh package, and it failed then.

Are you able to run the commands on your system fine without problems? Maybe we should try to use a Docker environment and test the commands there (to rule out problems due to system config)?

Yes. The only one that's failing on my side is the zlib. libiconv and libxml2 are compiling. It could be a good idea to try it in an isolated environment like Docker to see better the configuration problems and so on.

franramirez688 commented 11 months ago

Hi @powellnorma, any news on that? Could we close the issue?

powellnorma commented 11 months ago

Yes, can be closed. Thank you for the help

conan-io / conan