search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.23k stars 979 forks source link

[bug] during crossbuild, build requirements are buit for host profile instead of build profile #7565

Open madebr opened 4 years ago

madebr commented 4 years ago

When cross building a package and rebuilding all its dependencies, all its dependencies, including build requirements, are built using the host profile. It should use the build profile for the build requirements.

Environment Details (include every applicable attribute)

Steps to reproduce (Include if Applicable)

Logs (Executed commands with output) (Include/Attach if Applicable)

While rebuilding m4, the log contains:

m4/1.4.18: Calling:
 > source_subfolder/configure '--prefix=/home/maarten/.conan/data/m4/1.4.18/_/_/package/abd3ca9581f5ec3d6672fa2ee8818b1f09dbb082' '--bindir=${prefix}/bin' '--sbindir=${prefix}/bin' '--libexecdir=${prefix}/bin' '--libdir=${prefix}/lib' '--includedir=${prefix}/include' '--oldincludedir=${prefix}/include' '--datarootdir=${prefix}/share' --target=x86-linux-gnu

which means m4 is rebuilt for x86 (=host) instead of x86_64 (=build profile)

jgsogo commented 4 years ago

Working on my machine (after cloning your autoconf branch):

docker run -it -v$(pwd):/home/ubuntu/ conan/gcc9 bash
cd /home/ubuntu/
conan export recipes/autoconf/all/conanfile.py autoconf/2.69@

Let's check package IDs first:

arch=x86

conan info autoconf/2.69@ --profile=default -s=arch=x86 -n id
autoconf/2.69
    ID: 634482ae2711f292a960087dc3f4e0b99e4881dc
m4/1.4.18
    ID: 9efecfd5d5097a3c59d3dcb372f978c2c3d2cee6

arch=x86_64

conan info autoconf/2.69@ --profile=default -s=arch=x86_64 -n id
autoconf/2.69
    ID: 4db1be536558d833e52e862fd84d64d75c2b3656
m4/1.4.18
    ID: 2a30b7d6ea2202e5393ebda51c8729dd2162b9f8

And now see which one is used in the conan create command

conan create recipes/libcurl/all/conanfile.py libcurl/7.71.1@ --build=libcurl --profile:host=default --profile:build=default --settings:build=arch=x86

libcurl/7.71.1: Forced build from source
libcurl/7.71.1 (test package): Installing package
Requirements
    libcurl/7.71.1 from local cache - Cache
    openssl/1.1.1g from 'conan-center' - Cache
    zlib/1.2.11 from 'conan-center' - Cache
Packages
    libcurl/7.71.1:eb186ad30eac67be94e708b3027620138a2d5378 - Build
    openssl/1.1.1g:6af9cc7cb931c5ad942174fd7838eb655717c709 - Download
    zlib/1.2.11:6af9cc7cb931c5ad942174fd7838eb655717c709 - Download
Build requirements
    autoconf/2.69 from local cache - Cache
    automake/1.16.2 from 'conan-center' - Cache
    libtool/2.4.6 from 'conan-center' - Cache
    m4/1.4.18 from 'conan-center' - Cache
Build requirements packages
    autoconf/2.69:634482ae2711f292a960087dc3f4e0b99e4881dc - Missing
    automake/1.16.2:258f6f05ca54813b3d7180161753df7f9ccf7e1d - Download
    libtool/2.4.6:1216c48521e2cd3c4b31b9f4fd771694e1f03bc7 - Missing
    m4/1.4.18:9efecfd5d5097a3c59d3dcb372f978c2c3d2cee6 - Missing

Cross-build from 'Linux:x86' to 'Linux:x86_64'
Installing (downloading, building) binaries...
ERROR: Missing binary: autoconf/2.69:634482ae2711f292a960087dc3f4e0b99e4881dc
ERROR: Missing binary: libtool/2.4.6:1216c48521e2cd3c4b31b9f4fd771694e1f03bc7
ERROR: Missing binary: m4/1.4.18:9efecfd5d5097a3c59d3dcb372f978c2c3d2cee6

As you can see, it requires 634482ae2711f292a960087dc3f4e0b99e4881dc which is the package-ID corresponding to arch=x86 as we have forced with the parameters --profile:build=default --settings:build=arch=x86. At least settings are properly assigned, I haven't reviewed the recipe itself.

Note.- Pay attention, there is a bug in the output of the profile under Configuration (profile_build):, it doesn't update the arch value from the command line, but it is properly applied to the recipe.

I think I need more information from your output in order to know what is happening there.

madebr commented 4 years ago

autoconf is not really a problem, because it is arch independent. It is a collection of bash, m4 and perl scripts. I patched autoconf because its use of arch_build and os_build might clash with cross building. The issue shows up with m4, which is a native executable.

Output of conan create . libcurl/7.71.1@ -pr:h default32 -pr:b default64 --build "*" 2>&1 >output.log

Note the --target=x86-linux-gnu gnu triplet at line 110 for m4 and line 3218 for libtool.

I checked the m4 and libtool build folder in the conan prefix, and they are both built using 64-bit (-m64 flag). There was no -m32 flag present.

So it seems the gnu triplet is calculated wrongly.

jgsogo commented 4 years ago

The requested package-Id for m4 is correct as well, in my example it is using the 9efecfd... which correspond to x86 which is the architecture for the build context. So the error must be in the triplet calculation or in the recipe itself.

Computed triplet is --target=x86-linux-gnu. According to the m4 recipe it is calculated inside Conan, no value is overridden from the recipe, so the value is taken from the AutoToolsBuildEnvironment::_get_host_build_target_flags:

        if self._os_target and self._arch_target:
            try:
                target = get_gnu_triplet(self._os_target, self._arch_target, self._compiler)
            except ConanException as exc:
                self._conanfile.output.warn(str(exc))
                target = None
        else:
            target = None

The values fro self._os_target and self._arch_target are taken from the host context, they Linux and x86 respectively (from the POV of m4, it is generating binaries for the host context). From my understanding, a triplet like x86-linux-gnu is expected.

What should be the input to the configure script? Maybe the problem is that it is missing the --host argument to tell that the resulting binary of m4 should be x86? (I have very little experience with autotools).

madebr commented 4 years ago

@jgsogo Those are the canonical triplets, which is checked/generated by autoconf, and can be checked by the configure script. See https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/autoconf.html#Canonicalizing There exists triplets for: build, host and target. os/arch/compiler should be used for the host triplet (--host xxx), os_build/arch_build/compiler_build for the build triplet, os_target/arch_target/compiler_target for the target triplet.

For m4, only the host and build triplet matter, as there is no target. So _os_target and _arch_target should not be used.

See my gnu-config pr. That is adding scripts to detect and verify triplets.

k0ekk0ek commented 3 years ago

In my case I'm trying to get a 32-bit Windows build going on CI and I'm also running into this issue (I think). The build environment is 64-bit Windows. I'm using MSYS2 by setting CONAN_BASH_PATH and then simply invoke conan -b missing -s arch=x86 -s build_arch=x86_64 -s build_type=Debug cunit/2.1-3@.

The build_requirements lists libtool. And so libtool > automake > autoconf > m4 are required. Both libtool and m4 are built for x86, not automake and autoconf. To me it seems that somehow the build_requirements aren't propagated correctly. Since they're requirements for building, I'd expect conan to simply download the x86_64 versions and use those to compile cunit.

madebr commented 3 years ago

automake and autoconf don't contain binaries. So there is no difference between a 32-bit and 64-bit package. In these recipes, settings.arch and settings.compiler are not taken into account for the package id. https://github.com/conan-io/conan-center-index/blob/2f4af72088cb85eeea87868c908791022eb4fe71/recipes/autoconf/all/conanfile.py#L27-L31 Using the same autoconf package for both intel architectures should not cause problems.

k0ekk0ek commented 3 years ago

Entirely correct. autoconf and automake are simply downloaded. libtool and m4 are not. I'd expect these tools simply be download as I'm doing a cross build from x86_64 to x86.

madebr commented 3 years ago

libtool contains a library and m4 contains an executable.

libtool has os, arch, compiler, build_type in its settings (https://github.com/conan-io/conan-center-index/blob/2f4af72088cb85eeea87868c908791022eb4fe71/recipes/libtool/all/conanfile.py#L20), arch_build is not in it.

Your -s build_arch (or if you meant -s arch_build) does not make it cross build. You need to create a 32-bit profile and a 64-bit profile (see https://docs.conan.io/en/1.35/systems_cross_building/cross_building.html#cross-building-with-conan) and use -pr:b/-pr:h.

k0ekk0ek commented 3 years ago

That does the trick regarding the downloading of packages, thanks! However, that does lead to:

cunit/2.1-3: Calling build()
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.8.2
** Copyright (c) 2020 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x86'
cunit/2.1-3:
cunit/2.1-3: ERROR: Package '486ece6aed18ca03ed25554b84cacdd8db4d6674' build failed
cunit/2.1-3: WARN: Build folder C:\Users\k0ekk0ek\.conan\data\cunit\2.1-3\_\_\build\486ece6aed18ca03ed25554b84cacdd8db4d6674
ERROR: cunit/2.1-3: Error in build() method, line 117
        with self._build_context():
        AttributeError:

And that's caused by self.deps_user_info being empty (automake is the specific key it's looking for). Any idea what might be the issue there?

madebr commented 3 years ago

That's a bug in cunit not supporting cross building. Please open a ticket on the cci bug tracker. When cross building, a recipe also has self.user_info_build which should have been used here.

We're currently in the process of reworking a lot of recipes.

TedSmithNXC commented 3 years ago

@madebr : I am cross building on linux for arm. I am finding my m4 packages are not unique per host arch setting which is the problem you fixed above. I believe I am using your merged fixes for autoconf that should cause the gnu-triplet to be properly calculated. I am finding the m4 package being fetched is for the build arch of x86_64 instead of the host arch of armv7hf.

Do you have any guidance as to what I might look at next?

I just pulled conan-center-index and manually created and uploaded the pkgs to my artifactory using this hash:

commit 48043fa744b9ffa4db3336cfcd9b8c6b1ea57e2e (HEAD -> master, origin/master)
Author: SpaceIm <30052553+SpaceIm@users.noreply.github.com>
Date:   Mon Jul 26 23:04:16 2021 +0200

excerpts from my build

Configuration (profile_host):
[settings]
arch=armv7hf
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=8
os=Linux
os_build=Linux
[options]
[build_requires]
[env]
AR=arm-linux-gnueabihf-ar
AS=arm-linux-gnueabihf-as
CC=arm-linux-gnueabihf-gcc-8
CMAKE_CXX_COMPILER=arm-linux-gnueabihf-g++-8
CMAKE_C_COMPILER=arm-linux-gnueabihf-gcc-8
CONAN_CMAKE_TOOLCHAIN_FILE=/root/.conan/arm32v7_cmake.toolchain
CXX=arm-linux-gnueabihf-g++-8
FC=arm-linux-gnueabihf-gfortran-8
LD=arm-linux-gnueabihf-ld
RANLIB=arm-linux-gnueabihf-ranlib
STRIP=arm-linux-gnueabihf-strip
Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=7
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

...

Requirements
    libalsa/1.1.9 from 'nxc' - Downloaded
Python requires
    boost_base/1.69.0@bincrafters/stable
Packages
    libalsa/1.1.9:8becccfcd4d3af31e2ed07435c74e18c58a63c1d - Build
Build requirements
    autoconf/2.71 from 'my_art' - Downloaded
    automake/1.16.3 from 'nxc' - Downloaded
    gnu-config/cci.20201022 from 'nxc' - Downloaded
    libtool/2.4.6 from 'nxc' - Downloaded
    m4/1.4.18 from 'nxc' - Downloaded
Build requirements packages
    autoconf/2.71:cb054d0b3e1ca595dc66bc2339d40f1f8f04ab31 - Download
    automake/1.16.3:258f6f05ca54813b3d7180161753df7f9ccf7e1d - Download
    gnu-config/cci.20201022:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Download
    libtool/2.4.6:6fbe64f35b9d3eb3f534cbf1646451f40f718a31 - Build
    m4/1.4.18:24647d9fe8ec489125dfbae4b3ebefaf7581674c - Download

Cross-build from 'Linux:x86_64' to 'Linux:armv7hf'
Installing (downloading, building) binaries...

...

libtool/2.4.6: Copying sources to build folder
libtool/2.4.6: Building your package in /root/.conan/data/libtool/2.4.6/_/_/build/6fbe64f35b9d3eb3f534cbf1646451f40f718a31
libtool/2.4.6: Generator txt created conanbuildinfo.txt
libtool/2.4.6: Calling build()
libtool/2.4.6: Calling:
 > /root/.conan/data/libtool/2.4.6/_/_/build/6fbe64f35b9d3eb3f534cbf1646451f40f718a31/source_subfolder/configure '--datarootdir=/root/.conan/data/libtool/2.4.6/_/_/package/6fbe64f35b9d3eb3f534cbf1646451f40f718a31/bin/share' '--prefix=/root/.conan/data/libtool/2.4.6/_/_/package/6fbe64f35b9d3eb3f534cbf1646451f40f718a31' '--enable-shared' '--enable-static' '--enable-ltdl-install' '--bindir=${prefix}/bin' '--sbindir=${prefix}/bin' '--libexecdir=${prefix}/bin' '--libdir=${prefix}/lib' '--includedir=${prefix}/include' '--oldincludedir=${prefix}/include' --target=arm-linux-gnueabihf
## ------------------------- ##
## Configuring libtool 2.4.6 ##
## ------------------------- ##

checking for GNU M4 that supports accurate traces... configure: error: no acceptable m4 could be found in $PATH.
GNU M4 1.4.6 or later is required; 1.4.16 or newer is recommended.
GNU M4 1.4.15 uses a buggy replacement strstr on some systems.
Glibc 2.9 - 2.12 and GNU M4 1.4.11 - 1.4.15 have another strstr bug.
libtool/2.4.6: 
libtool/2.4.6: ERROR: Package '6fbe64f35b9d3eb3f534cbf1646451f40f718a31' build failed
libtool/2.4.6: WARN: Build folder /root/.conan/data/libtool/2.4.6/_/_/build/6fbe64f35b9d3eb3f534cbf1646451f40f718a31
ERROR: libtool/2.4.6: Error in build() method, line 104
        autotools = self._configure_autotools()
while calling '_configure_autotools', line 84
        self._autotools.configure(args=conf_args, configure_dir=self._source_subfolder)
        ConanException: Error 1 while executing /root/.conan/data/libtool/2.4.6/_/_/build/6fbe64f35b9d3eb3f534cbf1646451f40f718a31/source_subfolder/configure '--datarootdir=/root/.conan/data/libtool/2.4.6/_/_/package/6fbe64f35b9d3eb3f534cbf1646451f40f718a31/bin/share' '--prefix=/root/.conan/data/libtool/2.4.6/_/_/package/6fbe64f35b9d3eb3f534cbf1646451f40f718a31' '--enable-shared' '--enable-static' '--enable-ltdl-install' '--bindir=${prefix}/bin' '--sbindir=${prefix}/bin' '--libexecdir=${prefix}/bin' '--libdir=${prefix}/lib' '--includedir=${prefix}/include' '--oldincludedir=${prefix}/include' --target=arm-linux-gnueabihf
madebr commented 3 years ago

@TedSmithNXC It might be a bug in the libtool recipe. libtool needs m4, which is not (currently) a build requirement (only a requirement through automake -> autoconf -> m4).

Can you try applying the following patch to conan-center-index?

--- a/recipes/libtool/all/conanfile.py
+++ b/recipes/libtool/all/conanfile.py
@@ -53,6 +53,7 @@ class LibtoolConan(ConanFile):
         if tools.os_info.is_windows and not tools.get_env("CONAN_BASH_PATH"):
             self.build_requires("msys2/20200517")
         self.build_requires("gnu-config/cci.20201022")
+        self.build_requires("m4/1.4.18")

     @contextmanager
     def _build_context(self):
madebr commented 3 years ago

Anyway, I think this bug can be closed as my initial problem was not there all along.

kenfred commented 3 years ago

@madebr

libtool needs m4, which is not (currently) a build requirement (only a requirement through automake -> autoconf -> m4)

I'm trying to get my head around the host and build contexts and understand your answer here. So some of what I'm saying below might be flawed.

According to @TedSmithNXC's output, libtool is a build_requirement. So in a cross build, that means that, when building libtool, it should have the build-context: self.settings should equal the build profile (x86_64), self.setting_build should also equal the build profile, and self.settings_target should equal the host profile (armv7hf).

Since m4 is a requirement of libtool (via automake -> autoconf -> m4) it should share the same values for those settings.

@madebr suggests to make m4 a build requirement of libtool. If we do that, then the change I'd expect to see in the m4 recipe is for it to take self.settings_target to also be the build profile (x86_64). I believe this would essentially configure the m4 build like it is a native build on x86_64. @madebr is your theory that the cross-compiling libtool specifically needs a native m4? Can you explain this interaction?

I'm weak on autotools, so I don't fully understand what libtool and m4 need to run. libtool runs on the build system, but I believe libtool needs to know it is cross-compiling so it can help link up staged libraries in the target rootfs. Therefore, I would expect libtool should pay attention to its own self.settings_target and send that in as the --host argument when its build is being configured. According to conan's documentation, it tries to detect cross-compilations and populate the host argument. Is the issue that AutoToolsBuildEnvironment is not yet paying attention to self.settings_target?

kenfred commented 3 years ago

Is the issue that AutoToolsBuildEnvironment is not yet paying attention to self.settings_target?

Looking at the conan source code, I see that it is checking self.settings_target in AutoToolsBuildEnvironment.

madebr commented 3 years ago

When building libtool, the configure script prints the following:

## ------------------------- ##
## Configuring libtool 2.4.6 ##
## ------------------------- ##

checking for GNU M4 that supports accurate traces... /home/maarten/.conan/data/m4/1.4.18/_/_/package/24647d9fe8ec489125dfbae4b3ebefaf7581674c/bin/m4
checking whether /home/maarten/.conan/data/m4/1.4.18/_/_/package/24647d9fe8ec489125dfbae4b3ebefaf7581674c/bin/m4 accepts --gnu... yes
checking how m4 supports trace files... --debugfile

So yes, libtool needs m4 as a build requirement.

I downloaded a gcc-arm toolchain at ARM's website and could reproduce @TedSmithNXC 's problem.

It looks like there is something fundamentally wrong with autoconf and family.

Use the following default64 profile:

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[options]
[build_requires]
[env]
AS=gcc
ASFLAGS=-m64
ASM=gcc
ASMFLAGS=-m64
CC=gcc
CFLAGS=-m64
CXX=g++
CXXFLAGS=-m64
LDFLAGS=-m64

Run 

conan create recipes/autoconf/all autoconf/2.71@h -pr:b default64 -pr:b default64

Then m4 is not detected in autoconf's configure. Adding m4/1.4.18 to the build requirements of autoconf is no fix.

madebr commented 3 years ago

I've opened https://github.com/conan-io/conan-center-index/issues/6673 and https://github.com/conan-io/conan-center-index/pull/6674

This does not fix the problem, but perhaps we can pinpoint the cause better. I'm reopening this issue as I think again it is a problem with conan.

kenfred commented 3 years ago

@madebr Many thanks for digging into this!

So yes, libtool needs m4 as a build requirement.

I agree. But my understanding is that requirements are also available at the time of building, so there is no benefit to making something a build requirement if it is already a requirement. The only exception to this is if you need to change self.settings_target of that dependency, but I'm fuzzy on when that would be desired. Perhaps it is desired in this case?

You've proven this is not the fix, but I'm trying to solidify my understanding. :)

@TedSmithNXC Looks like @madebr is on to something over at https://github.com/conan-io/conan-center-index/issues/6673#issuecomment-892579356.

TedSmithNXC commented 3 years ago

@madebr Adding that build_requires of m4 to libtool does fix it. I understand now what is going on. Thank you again for your help!