search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.3k stars 985 forks source link

[question] Why do we skip 64/32 cross compilation with meson? #17261

Closed mattarroz closed 3 weeks ago

mattarroz commented 3 weeks ago

See https://github.com/conan-io/conan/blob/d35a73142a538808083f775ba3108664e1d50415/conan/tools/meson/toolchain.py#L159C2-L159C73

We are trying to cross build glib for linux x64->x86 (arch=x86) using the official conancenter recipe and the host profile

[settings]
os=Linux
arch=x86
compiler=clang
compiler.version=14
compiler.libcxx=libstdc++11
build_type=Debug

This leads to the following error: /usr/bin/ld: i386 architecture of input file '/conan/.conan/data/pcre2/10.42/_/_/package/9ff55d811dff94adb5ea6675dae30929f0465371/lib/libpcre2-8.a(pcre2_chartables.c.o)' is incompatible with i386:x86-64 output see glib_error_logs.txt

It can be fixed by setting tools.build.cross_building:cross_build=True, however shouldn't this work out of the box without any special configuration?

memsharded commented 3 weeks ago

Thanks for your question @mattarroz

In general, Windows compilation to x86 is not considered cross-compilation, but a native one, for other build systems. Maybe Meson is different here and needs to change that? Do you have any reference from Meson that could clarify this?

If that is the case, maybe the fix is to do cross_building(conanfile, skip_x64_x86=False) for Meson instead?

mattarroz commented 3 weeks ago

Thanks for your question @mattarroz

In general, Windows compilation to x86 is not considered cross-compilation, but a native one, for other build systems. Maybe Meson is different here and needs to change that? Do you have any reference from Meson that could clarify this?

Meson's cross-compilation documentation at least mentions x86 as an exemplary host architecture (see here).

If that is the case, maybe the fix is to do cross_building(conanfile, skip_x64_x86=False) for Meson instead?

Given that we had success explicitly overriding skip_x64_x86=True through tools.build.cross_building:cross_build=True, my guess is that setting it to False is the way to go. However, I am not a meson expert.

memsharded commented 3 weeks ago

Maybe @eli-schwartz can give here some feedback?

eli-schwartz commented 3 weeks ago

Meson isn't strongly opinionated about whether booting into x86_64 and running an i686 compiler to produce i686 binaries should be considered a cross build or a native build.

If you specify an i686 compiler using --cross-file= then meson will configure its cross architecture and detect a "native" x86_64 compiler for targets marked for native compilation (native: true, e.g. code generator tools written in C/C++ that do stuff like dump tables or data files in order to produce a header for the rest of the build). And it will use the i686 compiler to produce the installable artifacts.

If you specify an i686 compiler as the regular $CC then meson will simply regard your current build as an i686 userland running on an x86_64 kernel. This isn't even particularly unusual, there have been times when I installed an i686 linux distro and then manually installed their x86_64 kernel package while ignoring the configured system architecture, so I could execute 64-bit statically linked programs even though my operating system is i686. It's also how running an i686 container or chroot for CI purposes, works.

...

It is possible to have an x86_64 kernel that does NOT support i686 emulation. Your only option there would be configuring as a cross build.

memsharded commented 3 weeks ago

Thanks for the feedback @eli-schwartz

It seems that I was thinking mostly about Windows, but it is true that nowadays there are Linux systems that might not support the i686 modes, and for those, the current default assumption cross_building(conanfile, skip_x64_x86=True) wouldn't hold.

But it seems that changing it wouldn't be the best way to proceed, as it will definitely break users that depend on this behavior for their platforms that do support i686 mode, so it is possible that the recommended approach for this is to define the cross_build=True configuration explicitly (this is one of the reasons this configuration exists).

mattarroz commented 3 weeks ago

Thanks for the feedback @eli-schwartz

It seems that I was thinking mostly about Windows, but it is true that nowadays there are Linux systems that might not support the i686 modes, and for those, the current default assumption cross_building(conanfile, skip_x64_x86=True) wouldn't hold.

But it seems that changing it wouldn't be the best way to proceed, as it will definitely break users that depend on this behavior for their platforms that do support i686 mode, so it is possible that the recommended approach for this is to define the cross_build=True configuration explicitly (this is one of the reasons this configuration exists).

From an UX point of view, I believe we should improve here. Conan supports cross building by specifying build and host profile, so it seems very counter-intuitive still having to explicitly enable cross building by setting cross_build=True.

I know this might be an edge case, but let's improve usability and at least document this behavior. The documentation explicitly states skip_x64_x86=False as the default value: https://docs.conan.io/2/reference/tools/build.html#conan-tools-build-cross-building It should be mentioned that this isn't the case for meson, shouldn't it be? If that was the case, a good explanation on why the meson default value differs from other build tools should be given.

Maybe also a warning/info message should be implemented to advise the user that this setting is necessary for 64/32 bit cross builds.

anvesh202 commented 3 weeks ago

profile is changed accordingly:

[settings]
os=Linux
arch=x86
arch_build=x86_64
compiler=clang
compiler.version=14
compiler.libcxx=libstdc++11
build_type=Debug

[env]
CONAN_PROFILE_NAME=clang14_linux_x86_debug
CC=clang-14
CXX=clang++-14

#[buildenv]
#CFLAGS=-m32
#CXXFLAGS=-m32
#LDFLAGS=-m32

[conf]
tools.build.cross_building:cross_build=True

explicitly specifying conf to build for cross build, but flags were commented out. Meson is generating a cross file.

glib/2.81.0: Meson configure cmd: meson setup --cross-file "/conan/.conan/data/glib/2.81.0///build/1396172a72a68b0a2f8810cb73fc9927107ea0a3/build-release/conan/conan_mesoncross.ini" "/conan/.conan/data/glib/2.81.0///build/1396172a72a68b0a2f8810cb73fc9927107ea0a3/build-release" "/conan/.conan/data/glib/2.81.0//_/build/1396172a72a68b0a2f8810cb73fc9927107ea0a3/src" --prefix=/

detailed logs are attached in file. glib_32bit_logs.txt

Even after meson tool has generate a cross-file, confusing why still, 64-bit binaries are being generated? for reference, attaching cross file which is generated. conan_meson_cross.txt

But in commands of ninja there is no flag -m32 being added implicitly. In the [buildenv] if flags are mentioned explicitly, only then, the binaries are being built for 32 bit. why the flags are not being added implicitly once the cross build is recognized?

memsharded commented 3 weeks ago

From an UX point of view, I believe we should improve here. Conan supports cross building by specifying build and host profile, so it seems very counter-intuitive still having to explicitly enable cross building by setting cross_build=True.

The issue is that having build and host profile doesn't imply it is a cross-build. In Conan 2, the 2 build/host profiles are always used. And there are many cases with different build and host profiles that are not cross-build, for example:

So having different build-host profiles is not enough to guarantee that it is a cross-build.

It should be mentioned that this isn't the case for meson, shouldn't it be? If that was the case, a good explanation on why the meson default value differs from other build tools should be given.

That is indeed a relevant point. If the other build systems have other default, and Meson is not opinionated on this, I think it should be changed to align (could be considered a bug)

memsharded commented 3 weeks ago

Proposing https://github.com/conan-io/conan/pull/17266 to see if it breaks some exiting tests (would need some new tests to cover the change)

memsharded commented 3 weeks ago

This has been closed by https://github.com/conan-io/conan/pull/17266, it has fixed the default to align with other build systems, it will be released in next Conan 2.10